I am trying to find out where a cookie is being set.
I am running Varnish cache and want to know where the cookie is being set so I know if I can safely remove it for caching purposes.
The response headers look like this;
HTTP/1.1 200 OK
Server: Apache/2.2.17 (Ubuntu)
Expires: Mon, 05 Dec 2011 15:11:39 GMT
Cache-Control: no-store, max-age=7200
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
X-Session: NO
X-Cacheable: YES
Date: Tue, 04 Dec 2012 15:29:40 GMT
X-Varnish: 1233768756 1233766580
Age: 1081
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT
There is no cookie present. But when loading the same page in a browser the headers are the same, I get a cache hit and no cookie in the response headers.
But then the cookie is there all of a sudden, so it must be being somewhere. Even if I remove it it reappears. It even appears in Incognito mode in Chrome. But it is not in the header response.
I have been through all the javascript on the site and cannot find anything, is there any other way of setting a cookie?
Thanks.
If the Set-Cookie header goes through Varnish at some point, you can use varnishlog to find the request URL:
$ varnishlog -b -m 'RxHeader:Set-Cookie.*COOKIENAME'
This will give you a full varnishlog listing for the backend requests, including the TxURL to the backend which tells you what the client asked for when it got Set-Cookie back.
Related
I have a Django app served on EC2, and Cloudflare sits in front of the app to cache the static content (JS and CSS files; images are hosted separately on S3 and served via Cloudfront again with Cloudflare on top).
Every time I deploy new CSS, the hash in the filename of my CSS file changes, e.g. my_stylesheet.12345678.css.
Occasionally, after a release that involved a CSS change, I get some users emailing me that the website renders as "just text". Investigation led to find that the page in their browser has HTML that points to a previous release of the CSS file, e.g. my_stylesheet.11111111.css, which doesn't exist on my webserver anymore.
Since my website is very dynamic (it's a social network, so most pages will change at every request due to new posts, new comments, new likes, etc), to address this issue I have removed all client-side caching: I now send this header with pages like the main page:
cache-control: no-cache, no-store, must-revalidate, max-age=0.
I achieve this with Django's #never_cache decorator.
These are my Cloudflare page rules:
And these are the request/response header for the main page when I request it:
Request URL: https://www.example.com/
Request Method: GET
Status Code: 200
Remote Address: 111.111.111.111:443
Referrer Policy: strict-origin-when-cross-origin
cache-control: no-cache, no-store, must-revalidate, max-age=0
cf-cache-status: DYNAMIC
cf-ray: 63b9a4726dbe0e26-MXP
cf-request-id: 0947e51b7f00000e2661a28000000001
content-encoding: br
content-language: en
content-type: text/html; charset=utf-8
date: Tue, 06 Apr 2021 08:28:23 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
expires: Tue, 06 Apr 2021 08:28:23 GMT
nel: {"max_age":604800,"report_to":"cf-nel"}
report-to: {"max_age":604800,"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=YSpdY1LGNVOgbemDSBUL5DCg%2FDAX9NY3iGwLfpGvdQDQW9bxtzHIsNUxWFzei98JPs3IM2B4M%2FfepaN2CmIRH90HeUwOlYvCyNz%2Btd3iWavI"}],"group":"cf-nel"}
server: cloudflare
set-cookie: AWSALB=JcZVQeHr1BkaITzKti5bhIEeq9J9qZTufpjPTIvZukaeWLIEjlsOn75fFZXCIyMWN1F/NIYX2c7PsddmxAGNCSKp5EJxiZ59AKVTyOgVAfH89pqUPfX++uC3OUfF; Expires=Tue, 13 Apr 2021 08:28:22 GMT; Path=/
set-cookie: AWSALBCORS=JcZVQeHr1BkaITzKti5bhIEeq9J9qZTufpjPTIvZukaeWLIEjlsOn75fFZXCIyMWN1F/NIYX2c7PsddmxAGNCSKp5EJxiZ59AKVTyOgVAfH89pqUPfX++uC3OUfF; Expires=Tue, 13 Apr 2021 08:28:22 GMT; Path=/; SameSite=None; Secure
set-cookie: csrftoken=oUjzFDSxELNFfgtfNjqxIEkufxlyPaVaFXqsu7wK4jbK7xN1Z2ZQr5z0oDo9AYKO; expires=Tue, 05-Apr-2022 08:28:23 GMT; Max-Age=31449600; Path=/
set-cookie: sessionid=".eJyFWE1vG0cM_SuBz_Vg-DVfp8K596S7sJJWsRpZTrVS2iLIfy9Har2zq4ILGL6sOOSQ75GP8-NpfzgPl_X1dO673Xr_fl5f3r8dtmvwOfqn8oQe4dnTM-IKUpFUmJ1-YoxPv1i21NrKylMhX5hcDBgi2LY884uFU0F2zAGQbNvQ2vIKoXgpnF2MEdOCbWptaQW5QCwcnAacAtq2ubElWKHGDEWCo5SIxLAF73H97X24PBWSnIPlJbVeMK6ANS0FvEOUwHZWQ_rwErI3vcjMixQtPaFTF0DZ9JJ4VndNgqYxOS27eDvCFKZ-1VbTCOQoQAWUaQvTulesxkLsMIeMyc6__y8zQQ8wvcxQDb5gKj66CIEz2xHi7HZKh1CIXK4IWcgMfdQuIZtVnngJK6guCgaHlIO3axdiawsVXZyLeAdad7QwrLZhylmk4nPlDoJkqtw5dd_X28tf-rPr0J-twxI2hMBkus3TkNUtQaHkJIUgVtkrIaa2vt61wg1JYW7DLYcxwljJdDxsv9bvX_vT_b-eHfXuMQKHKOyV2vF504W4Zc6y2cnGd7WFnvs_rr16mRnFKDFFLQk8x-2eCYlS7pOk3S2wB3frmtT1aTjs9AwOrCeA__U3byOf5AP5KDw7t9tu-2FYH_vv_VEPrdZmTih-5CSy0mj3e3f68r4-6v9r96XXIw6Xhd4xohzMPhjHXhYDWTfEj0IFSNnm5-g9avjr7np5vad10201ITWz3XA5v28OJ1c_us_X4fL-9vLvV-twkYfR4rWl1rGU0G5tlGW8rEd76DZtwmzxMNYqhWglEMaGwhr8CpUiUBtKIkS7bQWMrS1XWxUBys7EzAstL8PIMJqW47UbXvVg2FASvyHZxbDdeYxd2Kad0qon2vq4Y9hsYxdtnZJz40bMKTEmN9-Q1HJlfz0eT91bBfnn1-587IdPL-duN7x2fy50uhYXeeVjEb5LDu0xXm2H69tbd_57vT8cL_15eCo_fppR8jjLLGZo-5PmPgYEahkbDpkAZJmqIJ0gKh48OT0lBXuCxDxG7o2mr3eMj5BElaYuYea4MOtDa3uLUCc4RycMSRZ02ux29xnng9PuHtnbttz6vWna-6BKOXCyqFCRN8uqXlaqLSnLeUlbPlQEQxF0GDyRpWnVttVU5G_glILggETY0hJVU03zrLZMVf8rZUkW1M6IzJRMTiKMyASzu4emu4v5yzhOoYgyG9Drod-ee_34RKHb61ymDjDC3u8XkNNg255CY9tL3uQvxZnium9JpFsSJVhQy6GZdfS_euLjorzBxEk6LVsOu009uOnFd71hiwxqoRBWOvh0rdJ1QiIx2bShCYykUl0UvuxAFWdaEjePfvlGGx0oJvR1LKRmLFiTNEwbihI73hY_r7cDbZcLgrtZyUwZ40OzopiUoHGZQbTaekaYAUjDptoKM2jB7daAMp3s_nZrFp1bwZuLqNrOVUHt3YWi89Gr6rVtwxQP6lcTrq2fKPu4EDNMx4ZOJsWSEkYCxYUVHf3DyMFcY1amBbMFK5Z8o7WM8ale8gPa9c-DE9Al1mYKpgdbHTBYH1yEiMdNo9-tv_Xnt8MwHN5Pw6Kwz-LTDOGiha5RoUpXc6hX7jcLht3KR9yCLcQwNQg3fllV7nQZVoTrQqvrevIpLmyH2U9t749B5PXW4mNY0K5TVaeCub5hqdonf1d11mY5Hbp1o6Xbo5uA7rQLj1_N8DDHTJZphPWdRW8HjlMMCxIKc5N_8zEip8c8QBUfVEFpP5fkPLO95QHEcVX9C49QuREQCyJ4Jo-gPsqIMi4KJbNSyTc7lJiyuNngVKDavbGR2kZu617W8MVegUfxAWy-pkizaVqdqt59yo761oPFi_ZSZYefP1FUqdCuR4f-0wt6MPJb28448qL5GjCqwCrDTYKMrVi1h_2wNP4SzDNDaLxb7YimpIPbG6XmDXRvUSFkUbvuPE25rchDKxWyOW5AHqSLLp9ETvevFGwFyQ2ks703N4u8RZMaTxM5__wHd3-R6w:1lTh4R:vHTnt7OdEA38hHpwyLYi3Ir0Vyg"; expires=Sun, 03-Oct-2021 08:28:23 GMT; httponly; Max-Age=15552000; Path=/
vary: Accept-Encoding, Accept-Language, Cookie
X-DNS-Prefetch-Control: off
:authority: www.example.com
:method: GET
:path: /
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: en-US,en-GB;q=0.9,en;q=0.8,it;q=0.7
cache-control: max-age=0
cookie: (redacted)
sec-ch-ua: "Google Chrome";v="89", "Chromium";v="89", ";Not A Brand";v="99"
sec-ch-ua-mobile: ?0
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: same-origin
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36
The issue is not specific to a browser, because I've had people report it about Chrome, people reporting about Firefox, people reporting about Safari.
One user had the same issue across all his browsers in his network, across multiple devices.
What could be causing this? How can I fix it?
Please let me know if you need more information to help me.
Thank you!
Changes to caching are not retroactive, meaning that if your users visited your website before you made the changes to the cache headers then the old cache control header would still be in effect.
In order to fix this, hope that your initial cache headers were setup to expire or refresh soon, otherwise users will need to manually clear their cache.
This also applies to proxies outside of your control that may have possibly cached your web page as well. Proxies inside your control should be purgeable.
Since you are using hashes in the filenames for resources you can leave the old files in place when you release a new version. This way your website won't break for users who still have old versions cached which refer to old versions of resources.
In order to have the latest version of the website load for users when they refresh the page, we use the following cache control for all index/html pages:
Cache-Control: public, must-revalidate, proxy-revalidate, max-age=0
So, I'm working with urllib2, and it keeps freezing on a specific page. Not even Ctrl-C will cancel the operation. It's throwing no errors (I'm catching everything), and I can't figure out how to break it. Is there a timeout option for urllib2 that defaults to never?
Here's the procedure:
req = urllib2.Request(url,headers={'User-Agent':'...<chrome's user agent string>...'})
page = urllib2.urlopen(req)
// p.s. I'm not installing any openers
Then, if the internet gets cut partway through the second line (which downloads it), even if connection is restored, this freezes the program completely.
Here's the response header I get in my browser (Chrome) from the same page:
HTTP/1.1 200 OK
Date: Wed, 15 Feb 2017 18:12:12 GMT
Content-Type: application/rss+xml; charset=UTF-8
Content-Length: 247377
Connection: keep-alive
ETag: "00e0dd2d7cab7cffeca0b46775e1be7e"
X-Robots-Tag: noindex, follow
Link: ; rel="https://api.w.org/"
Content-Encoding: gzip
Vary: Accept-Encoding
Cache-Control: max-age=600, private, must-revalidate
Expires: Wed, 15 Feb 2017 18:12:07 GMT
X-Cacheable: NO:Not Cacheable
Accept-Ranges: bytes
X-Served-From-Cache: Yes
Server: cloudflare-nginx
CF-RAY: 331ab9e1443656d5-IAD
p.s. The url is to a large WordPress feed which, according to the response, appears compressed.
According to the docs, the default timeout is, indeed, no timeout. You can specify a timeout when calling urlopen though. :)
page = urllib2.urlopen(req, timeout=30)
I am switching to Amazon Cloudfront for serving images on my website. To reduce load when we finally make it live, I thought of warming up the cache by hitting image URLs (I am making these request from India and expect majority of users to request from the same region so no need to have a copy of object on all edge locations worldwide).
The problem is that script uses curl to request image and when I access the same URL in browser I get MISS from Cloudfront. So Cloudfront is making two copies of object for these two request.
My current Cloudfront configuration forwards Content-Type request Header to origin.
How should I configure Cloudfront so that it doesn't care about request headers at all and once I made a request (whether curl or using browser) it should serve all future request for same resource from edge and not origin.
Request/Response headers-
I am afraid that the Cloudfront url won't be accessible from outside (until we go live) but I am posting request/response headers, this should give you fair idea. Also you can check out caching headers at origin - https://origin.ixigo.com/image/upload/t_thumb,f_auto/r7y6ykuajvlumkp4lk2a.jpg
Response after two successive request using browser
Remote Address:54.230.156.66:443
Request URL:https://youcannotaccess.com/image/upload/t_thumb,f_auto/r7y6ykuajvlumkp4lk2a.jpg
Request Method:GET
Status Code:200 OK
Response Headers
view source
Accept-Ranges:bytes
Age:23
Cache-Control:public, max-age=31557600
Connection:keep-alive
Content-Length:8708
Content-Type:image/jpg
Date:Fri, 27 Nov 2015 09:16:03 GMT
ETag:"-170562206"
Last-Modified:Sun, 29 Jun 2014 03:44:59 GMT
Vary:Accept-Encoding
Via:1.1 7968275877e438c758292828c0593684.cloudfront.net (CloudFront)
X-Amz-Cf-Id:fcbGLv8uBOP89qfR52OWa-NlqWkEREJPpZpy9ix0jdq8-a4oTx7lNw==
X-Backend:image6_40
X-Cache:Hit from cloudfront
X-Cache-Hits:0
X-Device:pc
X-DeviceType:pc
X-Powered-By:xyz
Now same url requested using curl but gave me miss
curl manu-mdc:cache manuc$ curl -I https://youcannotaccess.com/image/upload/t_thumb,f_auto/r7y6ykuajvlumkp4lk2a.jpg
HTTP/1.1 200 OK
Content-Type: image/jpg
Content-Length: 8708
Connection: keep-alive
Age: 0
Cache-Control: public, max-age=31557600
Date: Fri, 27 Nov 2015 09:16:47 GMT
ETag: "-170562206"
Last-Modified: Sun, 29 Jun 2014 03:44:59 GMT
X-Backend: image6_40
X-Cache-Hits: 0
X-Device: pc
X-DeviceType: pc
X-Powered-By: xyz
Vary: Accept-Encoding
X-Cache: Miss from cloudfront
Via: 1.1 4d42171c56a4c8b5c627040e6aa0938d.cloudfront.net (CloudFront)
X-Amz-Cf-Id: fY0LXhp7NlqB-I8F5-1TIMnA6bONjPD3CEp7dsyVdykP-7N2mbffvw==
Now this will give HIT
manu-mdc:cache manuc$ curl -I https://youcannotaccess.com/image/upload/t_thumb,f_auto/r7y6ykuajvlumkp4lk2a.jpg
HTTP/1.1 200 OK
Content-Type: image/jpg
Content-Length: 8708
Connection: keep-alive
Cache-Control: public, max-age=31557600
Date: Fri, 27 Nov 2015 09:16:47 GMT
ETag: "-170562206"
Last-Modified: Sun, 29 Jun 2014 03:44:59 GMT
X-Backend: image6_40
X-Cache-Hits: 0
X-Device: pc
X-DeviceType: pc
X-Powered-By: xyz
Age: 3
Vary: Accept-Encoding
X-Cache: Hit from cloudfront
Via: 1.1 6877899d48ba844a34ea4378ce336f06.cloudfront.net (CloudFront)
X-Amz-Cf-Id: qpPhbLX_5t2Xj0XZuZdjWD2w-BI80DUVyL496meQkLfSEn3ikt7hNg==
This is similar to this issue: Why are two requests with different clients from the same computer cache misses on cloudfront?
Depending on whether you provide the "Accept-Encoding: gzip" header or not, CloudFront edge server caches the object separately. Since browsers provides this header by default, and your site is likely to be accessed majorly via browser, I will suggest changing your curl call to include this header.
I was facing the same problem, after making the change in my curl call, I started to get a Hit from the browser on my first try via browser (after making a curl call).
Another thing I noticed is that CloudFront requires the full requested object to be downloaded before it will be cached. If you try to download the file partially by specifying the byte range in the curl, the intended object does not get cached, only the downloaded part gets cached as a different object. Same goes for a curl that was terminated in between. The other options I tried were wget call with spider option, but that internally does a HEAD call only and thus does not get the content cached on the edge server.
I'm developing a small site w/ Go and I'm trying to set a cookie from my server.
I'm running the server on localhost, with 127.0.0.1 aliased to subdomain-dev.domain.com on port 5080.
My When I receive the response for my POST to subdomain-dev.domain.com:5080/login I can see the set-cookie header. The response looks like this:
HTTP/1.1 307 Temporary Redirect
Location: /
Set-Cookie: myappcookie=encryptedvalue==; Path=/; Expires=Fri, 13 Sep 2013 21:12:12 UTC; Max-Age=900; HttpOnly; Secure
Content-Type: text/plain; charset=utf-8
Content-Length: 0
Date: Fri, 13 Sep 2013 20:57:12 GMT
Why isn't Chrome or Firefox recording this? In Chrome it doesn't show up in the Resources tab. In FF I can't see it either. And in neither do I see it in future Request headers.
See that Secure string in the cookie?
Yeah, me too. But only after a few hours.
Make sure you're accessing your site by SSL (https:// at the beginning of the URL) if you've got the Secure flag set.
If you're developing locally and don't have a cert, make sure you skip that option.
In my case, I had to add this to my response:
access-control-expose-headers: Set-Cookie
I found here that my Set-Cookie header was not accessible to my client unless I added it to the exposed-header header.
Hope this can help someone!
Found related github issue response cookies not being sent that helped.
In my case I am running react app under https (with mkcert tool) and making cross origin fetch request and get response. Cookies of the response is not set until I
specify credentials: 'include' for fetch request
example fetch api
fetch('https://example.com', {
credentials: 'include'
});
Specify these response headers from server
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: https://localhost:3000
Access-Control-Allow-Origin header has value of the url of my react app.
add these attributes of Set-Cookie Header Path=/; HttpOnly; Secure; SameSite=None using http cookies
Hope it helps someone!
I am setting up video delivery for video files to tv set-top boxes.
I want to use Amazon Cloudfront.
The video files are requested as usual http requets that may contain a range header to request partial resources (to enable the user on the box to jump into any position within the video).
My problem is that it is working on 2 of 3 boxes, one makes problems.
The request looks like this (sample data):
GET /path/file.mp4 HTTP/1.1
User-Agent: My User Agent
Host:myhost.com
Accept:*/*
Range: bytes=100-200
So if I do a request to cloudfront using telnet I see that the response is HTTP 1.0:
joe#flimmit-joe:~$ telnet d2zf9fl0izzsf6.cloudfront.net 80
Trying 216.137.61.164...
Connected to d2zf9fl0izzsf6.cloudfront.net.
Escape character is '^]'.
GET /skin/frontend/default/flimmit/images/headerbanners/02_green.png HTTP/1.1
User-Agent: My User Agent
Host:d2zf9fl0izzsf6.cloudfront.net
Accept:*/*
Range: bytes=100-200
HTTP/1.0 206 Partial Content
Date: Sun, 12 Feb 2012 18:42:15 GMT
Server: Apache/2.2.16 (Ubuntu)
Last-Modified: Tue, 26 Jul 2011 10:37:54 GMT
ETag: "1e0b8a-2d2b-4a8f6863ac11a"
Accept-Ranges: bytes
Cache-Control: max-age=2592000
Expires: Tue, 13 Mar 2012 18:42:15 GMT
Content-Type: image/png
Age: 351213
Content-Range: bytes 100-200/11563
Content-Length: 101
X-Cache: Hit from cloudfront
X-Amz-Cf-Id: W2fzPeBSWb8_Ha_UzvIepZH-Z9xibXyRddoHslJZ3TDXyFfjwE3UMQ==,CwiKc8-JGfE77KBVTTOyE9g-OYf7P-bCJZEWGwef9Es5rzhUBYKE8A==
Via: 1.0 972e3ba2f91fd0a38ea062d0cc03be37.cloudfront.net (CloudFront)
Connection: close
q�]#��ĥM�oӘ�i��i��������Y�.��/��ib���&
���
�Ⱦ�00�>�����Y`��X���r���s�=�n�s�b���7MConnection closed by foreign host.
joe#flimmit-joe:~$
Unfortunately I have only limited access to the box for testing purposes.
However this behavior by cloud-front seems strange to me so I wanted to ask whether it is even valid.
It is absolutely "valid" to answer with Http 1.0 to an Http 1.1 request.
I'll cite the Appendix 19.6 to the RFC2068 "It is beyond the scope of a protocol specification to mandate compliance with previous versions. HTTP/1.1 was deliberately designed, however, to make supporting previous versions easy."
http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.6
The important part is basically that the RFC does not force an Http 1.1 answer, so it's up to the server.