I'm trying to send an HTTP GET request in C++ using sockets, and I'm getting in response a 301 Moved permently, but to the same address I've asked for!
Here is my GET request :
GET /watch?v=1cQh1ccqu8M HTTP/1.1
Host: www.youtube.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Firefox/29.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language:en-US;q=0.6,en;q=0.4
Connection: keep-alive
All the \r\n are perfectly in place, because this GET request used to work for me not long ago, and I have not touched it since...
The response I'm getting from youtube :
HTTP/1.1 301 Moved Permanently
Date: Mon, 08 Dec 2014 11:04:10 GMT
Server: gwiseguy/2.0
Content-Type: text/html; charset=utf-8
Expires: Tue, 27 Apr 1971 19:44:06 EST
Location: https://www.youtube.com/watch?v=1cQh1ccqu8M
X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
Cache-Control: no-cache
Content-Length: 0
P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/bin/answer.py?answer=151657&hl=en for more info."
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic,p=0.002
According to the page they tell me to check http://support.google.com/accounts/bin/answer.py?answer=151657&hl=en, it says I need to add some kind of cookies now?
I've allways send this request without sending any cookies, so I am a bit confused...
me: Was your original request really done with https and not plain http...
#Amit: No, I was connecting to `www.youtube.com', then I've sent the GET request
Then you should look more closely at the redirect:
Location: https://www.youtube.com/watch?v=1cQh1ccqu8M
As you can see, this does redirect you to the same host, same page, but different protocol: you must use https instead of http.
Related
I'm using a fork of Django Simple SSO as an authentication mechanism on a Server.
It works great while we use the "standard" way to log in on the server. This means, we just access to the /login path, log in, and then navigate across the restricted paths, etc.
The problem comes when we want to connect to a 3rd server that also relays on SSO auth vía javascript/AJAX without direct access to /login path first.
I've tried to use django-cors-headers but didn't get the expected result.
I'm close to give up so any help is welcome.
The problem
Assume we are logged in on server 1 and auth server and we want a resource on server 2
When the browser is redirected to the Auth Server in order to verify that the browser is logged with a valid user, the cookie header is not transferred.
(See the diagram on button for further details on how Django Simple SSO works)
The headers set by the auth-server on the preflight request are the following:
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: accept-encoding, accept, authorization, cookie, content-type, dnt, origin, user-agent, x-csrftoken, x-grafana-org-id, x-dashboard-id, x-panel-id, x-requested-with
Access-Control-Allow-Methods: DELETE, GET, OPTIONS, PATCH, POST, PUT
Access-Control-Allow-Origin: https://server1-domain
Access-Control-Max-Age: 86400
Connection: keep-alive
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Fri, 16 Jul 2021 12:14:54 GMT
Server: nginx
Vary: Origin
The headers on the GET request where the following:
Request/Client:
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: es-ES,es;q=0.9,en-US;q=0.8,en;q=0.7
Connection: keep-alive
Host: auth-server-domain
Origin: https://server1-domain
sec-ch-ua: " Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"
sec-ch-ua-mobile: ?0
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36
x-grafana-org-id: 1
Response/Auth Server:
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: https://server1-domain
Connection: keep-alive
Content-Language: es
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Fri, 16 Jul 2021 12:14:54 GMT
Location: /error_unauthorized_page
Referrer-Policy: same-origin
Server: nginx
Vary: Accept-Language, Cookie, Origin
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
I think all the headers are correct, but it's clearly there are some error.
On the cookie side, I have tried to play with the SingleSite property, setting it to None and Lax without any change.
The domain set on the cookie browser's memory is "correct" from my point of view as it is .auth-server-domain (note the dot).
I'm using DJango 3.2.5 and Google Chrome as a web browser.
If you have any idea of why cookie is not being send on the auth-server-domain cors requests, I will really appreciate your comments.
If do you think I have missed any piece of relevant information, feel free to ask!
Thank everyone to reach this point and help me!
Appendix:
A simplified schema of how Simple SSO works:
I'm really really sorry for asking yet another CORS question but I'm stuck and I can't understand why.
I have any API that, on login, responds with a Set-Cookie header which sets a cookie with my user info. This is what that response looks like:
Response from api.somesite.com
Access-Control-Allow-Credentials: true
Connection: keep-alive
Content-Length: 183
Content-Type: text/html; charset=utf-8
Date: Mon, 19 Apr 2021 19:55:02 GMT
Set-Cookie: socialAccesstoken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6Ijk5MDk3NzQzNjkxNjU1MTY4MCIsInVzZXJuYW1lIjoiY3JlYXRpdmlpaSIsImRpc3BsYXlOYW1lIjoiTGVvIDjvuI_ig6MiLCJ1cmwiOiJodHRwczovL3QuY28vWjNBM01xaFBDbiIsInBob3RvIjoiaHR0cHM6Ly9wYnMudHdpbWcuY29tL3Byb2ZpbGVfaW1hZ2VzLzEzNzExMzQ3MjY0NzI0MTMxODYvZ0Q5Sk5lVU1fbm9ybWFsLmpwZyIsInByb3ZpZGVyIjoidHdpdHRlciIsImlhdCI6MTYxODg2MjEwMiwiZXhwIjoxNzA1MjYyMTAyfQ.Q7yWf-ywoZ-rWOhYseKTt_0V2_AlEMQ-cCL2rlRNm_U; Max-Age=86400; Path=/; Expires=Tue, 20 Apr 2021 19:55:02 GMT; HttpOnly
Vary: Origin
X-Powered-By: Express
The Cookie is set successfully and I can view it in the Application tab for api.somesite.com.
I then try to make a request from a page on the same domain:
Request from somesite.com to api.somesite.com
Response
HTTP/1.1 200 OK
X-Powered-By: Express
Access-Control-Allow-Origin: http://somesite.com:8000
Vary: Origin
Access-Control-Allow-Credentials: true
Token-Expired: true
Content-Type: application/json; charset=utf-8
Content-Length: 196
ETag: W/"c4-HQkmHNLuibgvd4gvz0JqjTLKjFU"
Date: Mon, 19 Apr 2021 19:59:18 GMT
Connection: keep-alive
Request
Host: api.somesite.com:3000
Connection: keep-alive
Content-Length: 143
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36
content-type: application/json
Accept: */*
Origin: http://somesite.com:8000
Referer: http://somesite.com:8000/
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: socialAccesstoken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6Ijk5MDk3NzQzNjkxNjU1MTY4MCIsInVzZXJuYW1lIjoiY3JlYXRpdmlpaSIsImRpc3BsYXlOYW1lIjoiTGVvIDjvuI_ig6MiLCJ1cmwiOiJodHRwczovL3QuY28vWjNBM01xaFBDbiIsInBob3RvIjoiaHR0cHM6Ly9wYnMudHdpbWcuY29tL3Byb2ZpbGVfaW1hZ2VzLzEzNzExMzQ3MjY0NzI0MTMxODYvZ0Q5Sk5lVU1fbm9ybWFsLmpwZyIsInByb3ZpZGVyIjoidHdpdHRlciIsImlhdCI6MTYxODg2MTQ2MiwiZXhwIjoxNzA1MjYxNDYyfQ.BpmQzWWYuBHLQQh6VpatflBsQUv2A0ahLqt1UgYOq8Q
As you can see the cookie is sent correctly.
But then if I switch my api to api.differentsite.com and make the same requests:
Response from api.differentsite.com
HTTP/1.1 200 OK
X-Powered-By: Express
Vary: Origin
Access-Control-Allow-Credentials: true
Set-Cookie: socialAccesstoken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6Ijk5MDk3NzQzNjkxNjU1MTY4MCIsInVzZXJuYW1lIjoiY3JlYXRpdmlpaSIsImRpc3BsYXlOYW1lIjoiTGVvIDjvuI_ig6MiLCJ1cmwiOiJodHRwczovL3QuY28vWjNBM01xaFBDbiIsInBob3RvIjoiaHR0cHM6Ly9wYnMudHdpbWcuY29tL3Byb2ZpbGVfaW1hZ2VzLzEzNzExMzQ3MjY0NzI0MTMxODYvZ0Q5Sk5lVU1fbm9ybWFsLmpwZyIsInByb3ZpZGVyIjoidHdpdHRlciIsImlhdCI6MTYxODg2MjYzMSwiZXhwIjoxNzA1MjYyNjMxfQ.MvOLarBw_Mz-h26WvOrOqyE0IaDQEnIqp2xLUiFqAZQ; Max-Age=86400; Path=/; Expires=Tue, 20 Apr 2021 20:03:51 GMT; HttpOnly
Content-Type: text/html; charset=utf-8
Content-Length: 183
Date: Mon, 19 Apr 2021 20:03:51 GMT
Connection: keep-alive
(the cookie is set correctly at api.differentsite.com:3000 according to the Application tab in Chrome)
And then I try to make a request from somesite.com:
Response
HTTP/1.1 200 OK
X-Powered-By: Express
Access-Control-Allow-Origin: http://somesite.com:8000
Vary: Origin
Access-Control-Allow-Credentials: true
Content-Type: application/json; charset=utf-8
Content-Length: 147
ETag: W/"93-2hkHY0W/lJkUQBK+JwrKp/OTCro"
Date: Mon, 19 Apr 2021 20:03:26 GMT
Connection: keep-alive
Request
POST /api HTTP/1.1
Host: api.differentsite.com:3000
Connection: keep-alive
Content-Length: 143
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36
content-type: application/json
Accept: */*
Origin: http://somesite.com:8000
Referer: http://somesite.com:8000/
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
There's no cookie sent, so authorisation fails.
Why is this happening? This should work, no?
My request looks like this:
const headers = new Headers();
headers.append("Content-Type", "application/json");
const result = await fetch(endpoint!, {
method: "POST",
headers,
body,
redirect: "follow",
credentials: "include",
}).then((response) => response.json());
I've checked over and over again and I'm certain these settings are correct.
I've followed the advice on this question but they didn't work. I'm unsure what Access-Control-Allow-Headers I should add on my request.
Looks like a night's sleep was all I needed.
The problem was caused by my cookies missing the correct settings. For httpOnly cookies to be sent in cross-domain requests they need to have:
same-site set to none
secure set to true
So I made sure these two values were set on cookies when in production:
sameSite: process.env.NODE_ENV === 'production' ? 'none' : 'lax',
secure: process.env.NODE_ENV === 'production' ? true : false,
And it started working.
EDIT:
While this is the correct answer, it seems that Safari now blocks third party cookies by default, meaning no cookies will be sent if the client is safari. This can be deactivated in the settings, but it's on by default.
I know about the SOAP -based webservices that SOAP messages, which are XML in turn, are transferred on the network, from client to the server. But what kind of data is transferred in case of RESTful webservice ?
from Wikipedia https://en.wikipedia.org/wiki/Representational_state_transfer
RESTful systems typically, but not always, communicate over the
Hypertext Transfer Protocol with the same HTTP verbs (GET, POST, PUT,
DELETE, etc.)
You can try yourself easily with curl or Fiddler. For example GitHub API is nice to experiment with.
Send this with Fiddler:
GET https://api.github.com/users/octocat HTTP/1.1
Host: api.github.com
User-Agent: Fiddler
and you will get this response:
HTTP/1.1 200 OK
Server: GitHub.com
Date: Fri, 10 Jul 2015 10:23:10 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1155
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 58
X-RateLimit-Reset: 1436527371
Cache-Control: public, max-age=60, s-maxage=60
Last-Modified: Mon, 06 Jul 2015 23:59:25 GMT
ETag: "d811d5844be3eaf9ab1f60dd36198aa9"
Vary: Accept
X-GitHub-Media-Type: github.v3; format=json
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: 3EAD7342:6BF1:E180441:559F9D0D
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: bd82876e9bf04990f289ba22f246ee9b
{"login":"octocat","id":583231,"avatar_url":"https://avatars.githubusercontent.com/u/583231?v=3","gravatar_id":"","url":"https://api.github.com/users/octocat","html_url":"https://github.com/octocat","followers_url":"https://api.github.com/users/octocat/followers","following_url":"https://api.github.com/users/octocat/following{/other_user}","gists_url":"https://api.github.com/users/octocat/gists{/gist_id}","starred_url":"https://api.github.com/users/octocat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/octocat/subscriptions","organizations_url":"https://api.github.com/users/octocat/orgs","repos_url":"https://api.github.com/users/octocat/repos","events_url":"https://api.github.com/users/octocat/events{/privacy}","received_events_url":"https://api.github.com/users/octocat/received_events","type":"User","site_admin":false,"name":"The Octocat","company":"GitHub","blog":"http://www.github.com/blog","location":"San Francisco","email":"octocat#github.com","hireable":false,"bio":null,"public_repos":5,"public_gists":8,"followers":1054,"following":6,"created_at":"2011-01-25T18:44:36Z","updated_at":"2015-07-06T23:59:25Z"}
The following link answered my query. Now I know that the data sent in case of RESTful services is 'raw http' data.
http://rest.elkstein.org/2008/02/how-simple-is-rest.html
I am using the Google Reports API to watch for changes to resources, such as a user's Google Calendar activity events. To achieve this, a channel (or watchpoint) was created by following the steps here: https://developers.google.com/admin-sdk/reports/v1/reference/activities/watch
Now, I would like to stop watching a resource before the channel expires. I sent a POST request along with a access token in the header and the body contains the channel id and the resource id (https://developers.google.com/admin-sdk/reports/v1/reference/channels/stop). However I keep getting a 404 Not Found. Am I doing something wrong here?
All other requests using the same access token are working fine.
My request/response looks like this:
POST /admin/reports/v1/channels/stop HTTP/1.1
Host: www.googleapis.com
Content-length: 97
Content-type: application/json
Authorization: Bearer ya29.rAExOpnO_gatfyJvKWEVt8OsQ-LyaCyN3UUFjYzm0-3ExEBZ9an7WWfdDLqJspChQaiiIQ
{
"id": "5cfc250b-2faf-4f86-91b3-398326c6b4fb",
"resourceId": "A_HZ7mQy0Zpd6-TkQjr3aQlWd94"
}
HTTP/1.1 404 Not Found
Content-length: 9
X-xss-protection: 1; mode=block
X-content-type-options: nosniff
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Vary: Origin,X-Origin
Server: GSE
Pragma: no-cache
Cache-control: no-cache, no-store, max-age=0, must-revalidate
Date: Fri, 10 Jul 2015 17:10:40 GMT
X-frame-options: SAMEORIGIN
Content-type: text/html; charset=UTF-8
Not Found
Turns out the request URL specified in the documentation is wrong.
The correct one is
https://www.googleapis.com/admin/reports_v1/channels/stop
Issue reported here: https://code.google.com/a/google.com/p/apps-api-issues/issues/detail?id=3914
I have web request that uses proxy (forward proxy) to website lets say www.example.com this web site has reverse proxy (Squid) as a result all my requests are returned as MISS.
Is there any way to use forward proxy and retrieve the data ?
Please understand that i am a newbie in this area.
After reading a while i set no cashing and this what i got
This is Request
Command: GET
URI: http://www.example.com
ProtocolVersion: HTTP/1.1
UserAgent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Referer: http://www.example.com
Accept: */*
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Keep-Alive: 115
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.7
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Host: www.example.com
Cookie:
PHPSESSID: 249477191de6048739cd8690fabd6060
UTG: A-3345389704b26912f6d5422.73487509-0a3a0a26a100a113a119a24a1a4a77a7a6a
addOtr: 7L4L2
CLIENT_WIDTH: 1916
MAIN_WIDTH: 1726
Cache-Control: no-store,no-cache
Pragma: no-cache
ProxyConnection: Keep-Alive
HeaderEnd: CRLF
This is Response
ProtocolVersion: HTTP/1.0
StatusCode: 407, Proxy authentication required
Reason: Proxy Authentication Required
Server: squid/3.0.STABLE19
Mime-Version: 1.0
Date: Mon, 31 Jan 2011 19:04:44 GMT
ContentType: text/html
ContentLength: 2986
X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
ProxyAuthenticate: Basic realm="Anonymous proxy"
Authenticate: Basic realm="Anonymous proxy"
X-Cache: MISS from funky
X-Cache-Lookup: NONE from funky:2448
Via: 1.0 funky (squid/3.0.STABLE19)
ProxyConnection: close
HeaderEnd: CRLF
Thanks in advance
You need to set your expires into the future.
Try changing your Expires, Cache-Control, and Pragma to look something like this:
Cache-Control:max-age=300
Date:Fri, 04 Feb 2011 04:52:58 GMT
Expires:Fri, 04 Feb 2011 04:57:58 GMT
...
(remove Pragma. You can do this by editing you .htaccess file)