I have a VB.NET app that sends a POST request to a script on my server that is running Cloudflare. I always get an error when sending the request from the app, however using a Firefox extension to simulate the request works fine. With the use of Fiddler I think I have found the cause of the problem:
When sending the request with the Firefox addon an extra header is attached to the request:
Cookie: __cfduidxxxxxxxxxxxx
This cookie is from Cloudflare, but where does it come from, ie. how can I get this cookie value and send it with my requests from the VB app? I tried copying and pasting the cookie into the app and it worked fine, so this leads me to conclude that I need this cookie, however this value is unique for each user so I cannot simply hardcode it into the app.
Quick side-note: Not sure if this helps, but if I send a GET request from the VB app it works fine without the __cfduid cookie.
Look for a Set-Cookie header coming back from the server on it's response. It will expect to get that value back on subsequent requests in a Cookie: header. This value is usually an opaque string that is classified by a path, although not always.
Related
i am playing around with Postman to get some insight on how things work behind the curtain and ran into, what I believe, is an issue but wanted to ask before I create a new issue on GitHub.
I am intercepting the request from my browser to the same site using the Postman Interceptor to use the request values in the native app. I have cookies enabled and the site (the whole domain) whitelisted.
When I use the history to resend the same request that was captured I get an auth error that is caused by the fact that the cookies are not included in the request (found that out by checking the cURL code snippet). I believe the reason for that is, that the cookies are set under another sub domain than that the request is send to.
I will try to include some pictures to clarify. My question here is:
Am I missing something/did I set something up in the wrong way
or is this an issue and I should create an issue in the official Postman Github page
cURL request
Cookies in Postman Native App
you should see if cookie is being send not using code snippet but the console :
its indeed sending cookies ,
This is a incognito window in chrome visiting oracle. Please notice that the request header already has cookie in the very request.
I also tried to use GuzzleHttp in php and postman. I can't get the cookie from anywhere.
Actually I am trying to crawl some other website, and that website has the same problem. I can't get the cookie so I got rejected.
Isn't cookie something that the server returns to the browser? Why in this case it is like the browse know the cookie in the first?
Http cookies are set once in a response by the server (with a Set-Cookie header), then they are included by the browser in each applicable subsequent request.
So it is perfectly normal that cookies your browser already obtained are present in the request but not in the response.
Cookies may also be set on browser side by Javascript, but that also can't happen before the first request (to at least retrieve that Javascript).
I'm working on an browser extension that authenticates with a remote server via XMLHttpRequests. In Firefox (59.0.2) I have the problem that the session cookie send by the server is not stored in the browser. When looking at the network traffic I get a Set-Cookie response from the server for every request:
Set-Cookie JSESSIONID=node01abks2u96hf84wt0i1uqwsb9879.node0;Path=/
but it seems that the cookie is never accepted or stored in the extension.
When looking at Chrome (where the extension is working) my extension includes this cookie in the request:
Cookie: io=jCX1X9rlaOhCqE0nAAAB JSESSIONID=node01abks2u96hf84wt0i1uqwsb9879.node0
However, this is not the case in Firefox. Why is Firefox is not including the cookie in the request? and why is it not storing the cookie?
UPDATE: as suggested I filed a bug report:
https://bugzilla.mozilla.org/show_bug.cgi?id=1454806
Furthermore, I created a very minimal example addon that fails:
https://gitlab.com/czeidler/firefox-cookie-problem
Could somebody please let me know if that addon really should work? or am I doing something wrong? To trigger the problem open the debug view of the addon and select the network view. Then click the addon popup icon. This will trigger two requests to my server. The first reply contains a Set-Cookie header that is not reused in the second request.
I found the reason why it is not working. Firefox handles a request from the popup as a cross domain request and does not set the cookie for this reason. Not sure if Chrome and Firefox should behave the same here or which approach is the better one. Here is how I fixed this issue to make it work in both browsers:
On the server:
response.addHeader("Access-Control-Allow-Origin", request.getHeader("Origin"))
response.addHeader("Access-Control-Allow-Credentials", "true")
In the popup:
connection.withCredentials = true;
I've run into a few problems with setting cookies, and based on the reading I've done, this should work, so I'm probably missing something important.
This situation:
Previously I received responses from my API and used JavaScript to save them as cookies, but then I found that using the set-cookie response header is more secure in a lot of situations.
I have 2 cookies: "nuser" (contains a username) and key (contains a session key). nuser shouldn't be httpOnly so that JavaScript can access it. Key should be httpOnly to prevent rogue scripts from stealing a user's session. Also, any request from the client to my API should contain the cookies.
The log-in request
Here's my current implementation: I make a request to my login api at localhost:8080/login/login (keep in mind that the web-client is hosted on localhost:80, but based on what I've read, port numbers shouldn't matter for cookies)
First the web-browser will make an OPTIONS request to confirm that all the headers are allowed. I've made sure that the server response includes access-control-allow-credentials to alert the browser that it's okay to store cookies.
Once it's received the OPTIONS request, the browser makes the actual POST request to the login API. It sends back the set-cookie header and everything looks good at this point.
The Problems
This set-up yields 2 problems. Firstly, though the nuser cookie is not httpOnly, I don't seem to be able to access it via JavaScript. I'm able to see nuser in my browser's cookie option menu, but document.cookie yeilds "".
Secondly, the browser seems to only place the Cookie request header in requests to the exact same API (the login API):
But, if I do a request to a different API that's still on my localhost server, the cookie header isn't present:
Oh, and this returns a 406 just because my server is currently configured to do that if the user isn't validated. I know that this should probably be 403, but the thing to focus on in this image is the fact that the "cookie" header isn't included among the request headers.
So, I've explained my implementation based on my current understanding of cookies, but I'm obviously missing something. Posting exactly what the request and response headers should look like for each task would be greatly appreciated. Thanks.
Okay, still not exactly what was causing the problem with this specific case, but I updated my localhost:80 server to accept api requests, then do a subsequent request to localhost:8080 to get the proper information. Because the set-cookie header is being set by localhost:80 (the client's origin), everything worked fine. From my reading before, I thought that ports didn't matter, but apparently they do.
I'm stuck in a cookie related question. I want to write a program that can automate download the attachments of this forum. So I should maintain the cookies this site send to me. When I send a GET request in my program to the login page, I got the cookie such as Set-Cookie: sso_sid=0589a967; domain=.it168.com in my program. Now if I use a cookie viewer such as cookie monster and send the same GET request, my program get the same result, but the cookie viewer shows that the site also send me two cookies which are:
testcookie http://get2know.it/myimages/2009-12-27_072438.jpg and token http://get2know.it/myimages/2009-12-27_072442.jpg
My question is: Where did the two cookie came from? Why they did not show in my program?
Thanks.
Your best bet to figure out screen-scraping problems like this one is to use Fiddler. Using Fiddler, you can compare exactly what is going over the wire in your app vs. when accessing the site from a browser. I suspect you'll see some difference between headers sent by your app vs. headers sent by the browser-- this will likley account for the difference you're seeing.
Next, you can do one of two things:
change your app to send exactly the headers that the browser does (and, if you do this, you should get exactly the response that a real browser gets).
using Fiddler's "request builder" feature, start removing headers one by one and re-issuing the request. At some point, you'll remove a header which makes the response not match the response you're looking for. That means that header is required. Continue for all other headers until you have a list of headers that are required by the site to yield the response you want.
Personally, I like option #2 since it requires a minimum amount of header-setting code, although it's harder initially to figure out which headers the site requires.
On your actual question of why you're seeing 2 cookies, only the diagnosis above will tell you for sure, but I suspect it may have to do with the mechanism that some sites use to detect clients who don't accept cookies. On the first request in a session, many sites will "probe" a client to see if the client accepts cookies. Typically they'll do this:
if the request doesn't have a cookie on it, the site will redirect the client to a special "cookie setting" URL.
The redirect response, in addition to having a Location: header which does the redirect, will also return a Set-Cookie header to set the cookie. The redirect will typically contain the original URL as a query string parameter.
The server-side handler for the "cookie setter" page will then look at the incoming cookie. If it's blank, this means that the user's browser is set to not accept cookies, and the site will typically redirect the user to a "sorry, you must use cookies to use this site" page.
If, however, there is a cookie header send to the "cookie setter" URL, then the client does in fact accept cookies, and the handler will simply redirect the client back to the original URL.
The original URL, once you move on to the next page, may add an additional cookie (e.g. for a login token).
Anyway, that's one way you could end up with two cookies. Only diagnosis with Fiddler (or a similar tool) will tell you for sure, though.