What I need:
I need to get all the currently applicable cookies for a page state. For example, I navigate to a site in the US, the browser has a lot of cookies populated by that page. I then decline all cookies on that site. Afterwards the browser reports much fewer cookies for that page. I need to be able to get that list of applicable cookies for the page as displayed in Chrome's View site information pop-out dialog left of the URL.
What I've tried:
I'm currently using Puppeteer, though I've also tried using Playwright. Both can utilize the Chrome Devtools Protocol (CDP) to gather information about a browser session and other related HTTP data.
I have a use case where I need to see the current state of cookies as it pertains to a web page. CDP provides two methods to accomplish this. Network.getAllCookies and Network.getCookies.
Network.getCookies returns cookies for a given URL, or else it provides all the cookies for the URL of the current page when no URL is provided. The problem with this is it's not a full picture. Network.getCookies does not include 3rd party cookies unless that domain is specifically requested. But there's no way for me to know all the 3rd party cookies for a given page.
If I switch it up and use Network.getAllCookies, that has too much information. It returns with all the cookies currently known to the browser session, regardless of applicability to the current page. For example, if I visit site A, then nav to site B, then I call Network.getAllCookies, I get all of the cookies from both pages since they are all in the same browser context. But I only want the 1st and 3rd party cookies for the current page I'm on.
In a headful browser it is possible to see all the applicable 1st and 3rd party cookies are for a site. If you click on the lock icon (secure site) to the left of the url, there is an option to view cookies. This is the most complete and accurate view of cookies I've come to know. Somehow the browser knows what 3rd party cookies originated from the current page and requests made by the current page and therefore displays them there. But how? And how can I get that list through CDP?
Note that document.cookie is useless as it does not include httpOnly cookies, and to my knowledge also doesn't return detailed cookie information (such as sameSite, secure, expiry, etc.)
I've tried using the CDP session directly attached to both of the above mentioned libraries. I've tried monitoring HTTP requests using Puppeteer and gathering set-cookie data from response headers, but CDP is unreliable when it comes to providing all the response header set-cookie information.
TLDR:
I'm expecting to programmatically retrieve only the relevant cookies for a page visit up to that point. If cookies become irrelevant in that page session, I don't want to receive them any more.
Network.getAllCookies returns too many cookies as it gives the whole browser context and not just the page relevant cookies.
Network.getCookies only returns cookies by a url.
document.cookie only returns application cookies.
How can I get all the 1st and 3rd party cookies that pertain to my current page?
I want this exact list:
Related
I have a server side application that uses cookies for session management. The browser has some script that sends an ajax request to add information to the session. This is working well and in production.
The business wants to be able to insert this application in other companies' websites via iframes. ie myapp.com is in an iframe in otherbusiness.com and when the user clicks a button in the application in the iframe launched from myapp.com, it sends a request with a cookie that contains the session id to update the user's session on the myapp.com server.
For the browser to be able to send a cookie, 3rd party cookies needs to be enabled by setting the cookie options of SameSite=None and Secure. This works for all browsers except Safari.
Safari no longer accepts 3rd party cookies.
The only solution I can come up with is to use session ids in the URL but this is a little cumbersome.
Can anyone suggest a better option or perhaps a good implementation of session ids in the url?
I used hidden html fields to pass the session id and expiration.
My server side code checks for a cookie if it cannot find it, looks for the session id and expiration in the hidden fields.
This avoids security issues with passing the id in the url. It is a little clumsy to implement but it works.
What are the components of a browser,Which are the settings that control a browser, how cookies works ,how browser sessions work?
Huge topic here. Rather than break this down and write forever, I will go through a web scenario: User types in address (or clicks a link?) - NOTE: a bit oversimplified
Browser breaks down URI
Browser checks cache to see if site IP address is in cache
If no, browser contacts DNS server to get IP address
Browser creates request for resource at URI, which is a package that has both a header (for routing) and a body (the request). For a page address typed in or click, it will be a GET request. The browser also sends a collection of "capabilities" like I accept cookies, etc.
Server contacted and returns a response.
Browser breaks down the response. It may be a success or failure and there will be a return code either way.
Assuming success, the browser then parses the message and breaks it into the HTML for the page and any collections sent (for example cookies)
For cookies, the browser checks user preferences before storing. It should be noted there is more than one type of cookie today. There are user cookies, which contain user information and can be easily blocked by the user and server cookies, which contain information needed by the application server. The later can also be blocked, if desired, but it is generally not advised as you lose functionality.
The HTML is parsed so the page can be displayed (rendering engine) and all resources required to see the page (like pictures) are requested through a new web request and rendered on the page.
Components? You can derive some here. Request creator, response parser, page renderer, configuration (both standard and user), etc.
Settings? Too numerous to cover. Open a browser and look at settings to see quite a few.
Cookies? Covered the basics already.
Sessions? Handled by server cookies. If you restrict them you can only get one page at a time, unless some information is passed in the URI on each request.
I'm trying to do a CORS GET that sends the cookie along with it. I've set all the headers (access-control-allow-origin, access-control-allow-credentials, access-control-allow-headers) in the server and am using withCredentials: true and crossDomain: true in the jquery ajax request. Everything works when I tell my browser to allow third-party cookies. Is there any way to do this without forcing visitors to allow third party cookies? I've even tried redirecting the user and redirecting back, but CORS will refuse to send the cookie along. :/
I've tried doing the CORS request via ajax, as well as via an iframe.
I don't think it is possible. See my (old but relevant) blog post on this.
The only bullet-proof way is to use 1st-party cookies (that is, open window in a top-level window like a separate tab, or redirect current window).
In some cases it is not necessary though. Browsers have slightly different notions of what third-party cookie is, and default behavior is also different. This post has a nice overview on these details. So in some cases you could do tricks to enable (or at least detect) use of cookies on the page.
Other workarounds include putting one server under a subdomain of the other (subdomains are usually not considered 3rd-party), or changing the flow so that the user is authenticated by other means than cookies.
if you set cookie (origin 2 set cookie) you should know third party cookie will be used, but if you only get cookie (only send cookie for origin 2 without setting in response) there is no reason third party cookie play a role.
so i think you set a cookie in origin 2 and because of that force you enable third party cookie.
Note that cookies set in CORS responses are subject to normal
third-party cookie policies. In the example above, the page is loaded
from foo.example but the cookie on line 19 is sent by bar.other, and
would thus not be saved if the user's browser is configured to reject
all third-party cookies.
Cross-Origin Resource Sharing mozilla.org
I created a Google site page with 5 links on it. Is it possible to create on my site a script or something that stores in a cookie the link on which the user has clicked, and then the next time he will connect to the page, he will be automatically redirected to the link he clicked on ? For information, the user connect to the site with his Google email account.
How can I do that please?
Thank you very much in advance for your help
While it is possible to read cookies and redirect using JavaScript inside a Google Page (using widgets), browsers will not allow you to set cookies for a completely different domain for obvious security reasons.
Related:
How to set a cookie for another domain
Cross-Domain Cookies
What's your favorite cross domain cookie sharing approach?
You could theoretically try and send an AJAX request from the Google Page with a "where should I direct this user to?" and expect a URL or a null.
See:
CORS $.ajax session cookies (access-control-allow-credentials & withCredentials=true)
Cross domain POST request is not sending cookie Ajax Jquery
But overall, your task is not as straightforward as it may seem. The browser will, fortunately, not play along.
On websites where you have to enter a user name and password, I notice that I can browse the site with one browser and it will know who I am no matter where I go on the site. But if I open a different browser it doesn't know who I am in that browser unless I log on in that browser.
After I log in to a website, does it store some kind of cookie in my browser, and every time I navigate to a different page on that site, it checks the cookie for my identity?
What would happen if I logged in, and then before browsing to a different page on the site, deleted the cookie?
This is more of a "teach a man to fish" answer, so I apologise if it's not what you were after. But if you take my advice you will learn lots, so please trust me :)
There's a number of tools that you can use to track exactly what http traffic is going between your browser and the server. One is called Firebug, a plugin for Firefox. The other kind of tool is called a "web debugging proxy". There's charles, which is very powerful, and fiddler, which is free.
What you want to do with any of these tools is use a website, and then look at the raw request. This shows you exactly what your browser is saying to the server. You'll see the cookies for that server are sent along with every request. What's cool about these tools is that you can edit a request just before it's sent, so you can test how the servers respond...
After I log in to a website, does it store some kind of cookie in my browser, and every time I navigate to a different page on that site, it checks the cookie for my identity?
Yes. The cookie is sent with each HTTP request.
What would happen if I logged in, and then before browsing to a different page on the site, deleted the cookie?
The same as if you were to switch browsers.
Every time when you navigate a new page, your browser sends a request to the server and the server sends back you the response. Your request contains the cookies, which the server can parse and use. You if you delete the cookie, your browser can't send it with the next request.
What would happen if I logged in, and then before browsing to a different page on the site, deleted the cookie?
You would no longer be logged in.
After I log in to a website, does it store some kind of cookie in my browser, and every time I navigate to a different page on that site, it checks the cookie for my identity?
Yes. Most likely, you are dealing with a "session-cookie". These cookies do not store any information themselves, but use a long string to identify yourself to a server. I would suggest doing some research on cookies. As for the (I'm guessing assumed) question of "Why cookies work on different pages?" is because cookies are tied to the domain, and not the exact URI.
Cookies contain names, values, and expirations (along with a few others). The most common you'll see are sessions, which use an identifier to load a session-state from the server containing your information. These are the safest cookies as everything is centralized and not as prone to hijacking. The other kind is a regular cookie, which has a limited size and stores information client-side. Anything that has to do with shopping or anything that tracks users most likely uses sessions, while something like a customizable javascript-y page probably uses a normal cookie. The former tracks information server-side for additional security, while the latter poses no security risk, and leaves the information for the client to manage.