I have web request that uses proxy (forward proxy) to website lets say www.example.com this web site has reverse proxy (Squid) as a result all my requests are returned as MISS.
Is there any way to use forward proxy and retrieve the data ?
Please understand that i am a newbie in this area.
After reading a while i set no cashing and this what i got
This is Request
Command: GET
URI: http://www.example.com
ProtocolVersion: HTTP/1.1
UserAgent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Referer: http://www.example.com
Accept: */*
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Keep-Alive: 115
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.7
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Host: www.example.com
Cookie:
PHPSESSID: 249477191de6048739cd8690fabd6060
UTG: A-3345389704b26912f6d5422.73487509-0a3a0a26a100a113a119a24a1a4a77a7a6a
addOtr: 7L4L2
CLIENT_WIDTH: 1916
MAIN_WIDTH: 1726
Cache-Control: no-store,no-cache
Pragma: no-cache
ProxyConnection: Keep-Alive
HeaderEnd: CRLF
This is Response
ProtocolVersion: HTTP/1.0
StatusCode: 407, Proxy authentication required
Reason: Proxy Authentication Required
Server: squid/3.0.STABLE19
Mime-Version: 1.0
Date: Mon, 31 Jan 2011 19:04:44 GMT
ContentType: text/html
ContentLength: 2986
X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
ProxyAuthenticate: Basic realm="Anonymous proxy"
Authenticate: Basic realm="Anonymous proxy"
X-Cache: MISS from funky
X-Cache-Lookup: NONE from funky:2448
Via: 1.0 funky (squid/3.0.STABLE19)
ProxyConnection: close
HeaderEnd: CRLF
Thanks in advance
You need to set your expires into the future.
Try changing your Expires, Cache-Control, and Pragma to look something like this:
Cache-Control:max-age=300
Date:Fri, 04 Feb 2011 04:52:58 GMT
Expires:Fri, 04 Feb 2011 04:57:58 GMT
...
(remove Pragma. You can do this by editing you .htaccess file)
Related
I am getting the following response from the Django (2.2) default login view:
Request URL: https://api.n.exchange/en/accounts/login/?next=/en/referrals/
Request Method: GET
Status Code: 200 OK
Remote Address: 104.25.23.99:443
Referrer Policy: no-referrer-when-downgrade
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
CF-RAY: 51105b439e71b50e-VNO
Connection: keep-alive
Content-Encoding: br
Content-Language: en
Content-Type: text/html; charset=utf-8
Date: Wed, 04 Sep 2019 13:37:09 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Expires: Wed, 04 Sep 2019 13:37:09 GMT
Server: cloudflare
Set-Cookie: csrftoken=BHfEypgp6ux4FvQr14G06DQnqHjRL0tXZYP4Cg2b67naaFkxFw29g0C5UVettETb; expires=Wed, 02 Sep 2020 13:37:09 GMT; Max-Age=31449600; Path=/; SameSite=Lax
Transfer-Encoding: chunked
Vary: Cookie, Origin
X-Frame-Options: SAMEORIGIN
X-NewRelic-App-Data: PxQGUlFVCwoGR1JTDwQFX1IAFB9AMQYAZBBZDEtZV0ZaCldOdxRdARBfWA9JB1JSXgMOTFReWRIWWFQdAxMXCh4UUQdPSw5+XAJQD2cIVhVKUVIVRE8IHwBKUVAPBw5QVggOBltfUVYDUw5WFBUFHhFVAFAABABbAQEGWFYGWQVSRk0EVl1EAzk=
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9,en-US;q=0.8,he;q=0.7,lt;q=0.6,de;q=0.5
Connection: keep-alive
Cookie: __cfduid=d76f7b7d2a1caa6948456ad6829dc25991553698344; _ga=GA1.2.2123122031.1553698346; _ym_uid=1553698347983819119; _ym_d=1553698347; crisp-client%2Fsession%2F6eb9ed9e-2c8b-48e8-a0ce-62c3ce81fb61=session_76921095-b26c-4790-a968-82cf111e3940; _hjid=e834477e-35c2-4ef9-aacd-5fb2d644ae2c; crisp-client%2Fsocket%2F6eb9ed9e-2c8b-48e8-a0ce-62c3ce81fb61=1; _gid=GA1.2.1927749960.1567447617; USER_TZ=Europe/Vilnius; django_language=en; _ym_isad=1; _ym_visorc_42222484=w; _ym_visorc_45642111=w; csrftoken=BHfEypgp6ux4FvQr14G06DQnqHjRL0tXZYP4Cg2b67naaFkxFw29g0C5UVettETb; _gat=1
Host: api.n.exchange
Referer: https://api.n.exchange/en/accounts/login/?next=/en/referrals/
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36
X-NewRelic-ID: VQUAV1VaDhADVVlXBQgBVw==
next: /en/referrals/
As you can clearly see, the set-cookie header for the sessionid which represents the authenticated Django session is missing.
What could be the cause? (at first, I was thinking the reason is that we have a self signed HTTP certificate behind Cloudflare but we have migrated to a valid letsencrypt certificate and removed cloudflare to test it, but the problem persists).
Thanks!
It seems that SESSION_COOKIE_DOMAIN was set to the wrong setting
I have sent below request to iCloud calendar service to get all calendar list. But response is always empty. Can anyone help me?
Request:
GET /10232836851/calendars/F41F7478-4345-4A4A-8CD5-548122EF2C22/ HTTP/1.1
HOST: pxx-caldav.icloud.com
user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2
authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXX==
depth: 1
content-type: text/xml
Response Header:
Server: AppleHttpServer/a6f3179
Date: Fri, 25 Nov 2016 07:35:53 GMT
Content-Length: 0
Connection: keep-alive
Last-Modified: Wed, 23 Nov 2016 19:32:45 GMT
Dav: 1, access-control, calendar-access, calendar-schedule, calendar-auto-schedule, calendar-managed-attachments, calendarserver-sharing, calendarserver-subscribed, calendarserver-home-sync
Accept-Ranges: bytes
X-Responding-Server: pv41p47ic-tydg09053001 23 a63660a6f7d1a25b5a7ed66dab0da843:44702101
X-Transaction-Id: 55b238a2-548b-41fc-acc0-b697adb4ab84
Cache-Control: private, max-age=0, no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Via: icloudedge:hk02p00ic-ztde010805:7401:16G8:Hong Kong
X-Apple-Request-Uuid: 55b238a2-548b-41fc-acc0-b697adb4ab84
Access-Control-Expose-Headers: X-Apple-Request-UUID; Via
To get the list of calendars on the server using the CalDAV protocol you use the PROPFIND method, not a GET. This is pretty well described in the SabreDAV Building a CalDAV Client web page.
Something like this:
PROPFIND /calendars/johndoe/ HTTP/1.1
Depth: 1
Content-Type: application/xml; charset=utf-8
Host: ...
Authorization: ...
<propfind xmlns="DAV:">
<prop>
<displayname />
</prop>
</propfind>
I'm trying to send an HTTP GET request in C++ using sockets, and I'm getting in response a 301 Moved permently, but to the same address I've asked for!
Here is my GET request :
GET /watch?v=1cQh1ccqu8M HTTP/1.1
Host: www.youtube.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Firefox/29.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language:en-US;q=0.6,en;q=0.4
Connection: keep-alive
All the \r\n are perfectly in place, because this GET request used to work for me not long ago, and I have not touched it since...
The response I'm getting from youtube :
HTTP/1.1 301 Moved Permanently
Date: Mon, 08 Dec 2014 11:04:10 GMT
Server: gwiseguy/2.0
Content-Type: text/html; charset=utf-8
Expires: Tue, 27 Apr 1971 19:44:06 EST
Location: https://www.youtube.com/watch?v=1cQh1ccqu8M
X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
Cache-Control: no-cache
Content-Length: 0
P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/bin/answer.py?answer=151657&hl=en for more info."
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic,p=0.002
According to the page they tell me to check http://support.google.com/accounts/bin/answer.py?answer=151657&hl=en, it says I need to add some kind of cookies now?
I've allways send this request without sending any cookies, so I am a bit confused...
me: Was your original request really done with https and not plain http...
#Amit: No, I was connecting to `www.youtube.com', then I've sent the GET request
Then you should look more closely at the redirect:
Location: https://www.youtube.com/watch?v=1cQh1ccqu8M
As you can see, this does redirect you to the same host, same page, but different protocol: you must use https instead of http.
I have a web server that is serving files. Here is a request I make and the answer:
GET / HTTP/1.1
Host: 127.0.0.1:8004
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20131030 Firefox/17.0 Iceweasel/17.0.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cookie: djdt=hide; sessionid=1ksbrjhfunsjtywlk2q4p2rl7zrsyvza; messages="Nice and all"
Cache-Control: max-age=0
HTTP/1.0 200 OK
Set-Cookie: csrftoken=Xkg4CnLEahplJPHEJGLrmGvPjebMqyh9; expires=Mon, 09-Feb-2015 21:15:51 GMT; Max-Age=31449600; Path=/, sessionid=qy5guifmpkra78ubkztxxlmk3tu5vr7s; expires=Mon, 24-Feb-2014 21:15:51 GMT; httponly; Max-Age=1209600; Path=/, messages=; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/
Vary: Cookie
Server: gunicorn/18.0
Date: Mon, 10 Feb 2014 21:15:51 GMT
x-frame-options: SAMEORIGIN
Content-Type: text/html; charset=utf-8
Up to that point everything is fine. Things get weird on the very next request:
GET /static/css/bootstrap.css HTTP/1.1
Host: 127.0.0.1:8004
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20131030 Firefox/17.0 Iceweasel/17.0.10
Accept: text/css,*/*;q=0.1
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Referer: http://127.0.0.1:8004/
Cookie: djdt=hide; sessionid=1ksbrjhfunsjtywlk2q4p2rl7zrsyvza; messages="Nice and all"
If-Modified-Since: Sun, 13 Oct 2013 20:23:36 GMT
Cache-Control: max-age=0
HTTP/1.0 304 NOT MODIFIED
Date: Mon, 10 Feb 2014 21:15:54 GMT
Server: WSGIServer/0.1 Python/2.7.3
Content-Length: 0
Yeah, the browser (Iceweasel) just sent the previous cookies, instead of sending the new one. Actually, it does not take the set-cookie HTTP header into account. That I don't understand. Same behavior on Chromium. On both browsers, nothing appears in the console. When using the developers tools from Iceweasel to the see the request, the previous cookie is parsed in the 'sent cookie', but the received cookie isn't parsed as it should be, such as it is in this (otherwise unrelevant) capture:
You may have notice the Server HTTP response header is different when serving the static file. Indeed, the local Django app serves as a proxy when requesting pages, but serves static content himself. Both contents are on the same domain, same port, so this is supposed to be the same from the browser perspective.
I have a FB app, which uses FB.api() from its JS SDK to post a message on user's feed. The issue is, I get 400 error instead.
(Please note many data have been modified for privacy)
The request was sent to URL:-
https://graph.facebook.com/me/feed/?access_token=178486488887736|2.AQCDILuiyM-wBx8.3600.1313269200.1-1000|_z6EB7Ebp6Bxgf-ss
Request header:-
Host: graph.facebook.com
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:5.0.1) Gecko/20100101 Firefox/5.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
DNT: 1
Connection: keep-alive
Cookie: datr=IezG1St6azJsvDL; lu=wgWUTDTuiTLJXgPI8A; locale=en_US; app_id=178486488887736; s=Aa7YfcM3PxNBg; L=2; act=1313261419%2F1; presence=EM31p_5f1B01179550304F5122K0H0U0OQ0EsF0CEblFDacF0EutF0PCC; c_user=100001304; sct=1313128632; xs=60%3Aa1c1753e6a6abd0fe7b23bfe4; p=167; _e_zzrj_3=%5B%22zzrj%22%2C1313261446%2C%22act%222Fyourmemorablestatus%2F%22%2C%7B%22ft%22%3A%7B%7D%2C%22gt%22%3A%7B%22tti_ms%22%3A5464%2C%22app_id%22%3A178486488887736%2C%22is_early_flush%22%3Afalse%2C%22browser%22%3A%22Firefox%205%22%7D%7D%2C0%2C16%5D
The request used POST method. Post data was:-
Referer: http://connect.facebook.net/rsrc.php/v1/yK/r/RIxWozDt5Qq.swf
Content-type: application/x-www-form-urlencoded
Content-length: 1993
access_token=1784864.1-1000011Ebp6Bxgf-ssENEpia7iPE&message=My%20memorable%20status%20messages%20%3A-)%0A%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%0A%0A%EF%BF%BD%20blog.applegrew.com%20and%20cink.applegrew.com%20both%20are%20now%20down.%20Moved%20them%20today%20to%20a%20new%20%22cloud%22%20based%20system%2C%20and%20now%20they%20are%20up%20in%20the%20air%20somewhere.%20Raised%20tickets%20against%20my%20service%20provider.%0A%0A%EF%BF%BD%20Check%20out%20the%20google%20home%20page%20today%2C%20and%20pluck%20the%20strings!%20%3A)%0A%0A%EF%BF%BD%20wow%20mallu%20girls%20r%20beautiful.%0A%0A%EF%BF%BD%20My%20Ctrl%2C%20C%2C%20V%20keys%20have%20started%20to%20wear%20out.%0A%0A%EF%BF%BD%20Developers%20are%20users%20too.%20People%20tend%20to%20forget%20that.%20%3A(%0A%0A%EF%BF%BD%20Once%20a%20wise%20one%20said%3A%20Bugs%20can%20neither%20be%20created%20nor%20destroyed%2C%20it%20has%20always%20existed%20and%20will%20come%20around%20in%20different%20manifestations.%0A%0A%EF%BF%BD%20unzip%2C%20strip%2C%20touch%2C%20finger%2C%20grep%2C%20mount%2C%20FSCK%2C%20more%2C%20yes%2C%20unmount%2C%20sleep%20.....%0A%0AThese%20are%20all%20Linux%20commands.%0A%0A%EF%BF%BD%20%EF%BF%BD%20Tonight%20going%20to%20Bryan%20Adam%20concert!%0A%0A&pretty=0&sdk=joey
Response header:-
Cache-Control: no-store
Content-Type: text/javascript; charset=UTF-8
Expires: Sat, 01 Jan 2000 00:00:00 GMT
P3P: CP="Facebook does not have a P3P policy. Learn why here: http://fb.me/p3p"
Pragma: no-cache
WWW-Authenticate: OAuth "Facebook Platform" "invalid_request" "(#1) An unknown error occurred"
X-FB-Rev: 422152
Set-Cookie: _e_zzrj_3=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.facebook.com; httponly
X-FB-Server: 10.62.1.41
X-Cnection: close
Date: Sat, 13 Aug 2011 19:43:26 GMT
Content-Length: 78
Well I found the problem. In FB message can be max 420 characters long (including spce etc.). I wish FB responded with a more humane error.