Browser not setting cookie (set-cookie is in the header) - Django backend, Express fetch frontend - django

I have a Django/DRF API backend that I'm POSTing login credentials to and expecting a "sessionid" cookie in return. It's running at https://url.com/api/. The login endpoint is https://url.com/api/api_login/.
I'm using ExpressJS and fetch on the frontend to make the call. It's running at https://url.com/. The login form is located at https://url.com/login.
I have an Nginx reverse proxy mapping "url.com/api" to "url.com:8002", and "url.com" to "url.com:8003".
Here is the simplified code for the backend:
# views.py
#method_decorator(csrf_exempt, name='dispatch')
class ApiLogin(APIView):
def post(self, request):
form = LoginForm(request.POST)
if form.is_valid():
user = authenticate(request, username=form.cleaned_data['username'], password=form.cleaned_data['password'])
if user is not None:
auth_login(request, user)
# at this point, you are either logged in or not
if request.user.is_authenticated:
response = HttpResponse(f"Successful login for {form.cleaned_data['username']}.")
return response
else:
response = HttpResponse("Login failed.")
return response
Here is the full code for the frontend:
//*** server.js
const express = require('express');
const app = express();
app.use(express.static('static'));
app.use(express.json());
app.use(express.urlencoded({extended: true}));
const router = require('./router');
app.use(router);
//*** router.js
const express = require('express');
const router = express.Router();
const qs = require('qs');
const fetch = require('node-fetch');
// temporarily running on a self-signed cert, so this bypasses the cert-check
const https = require('https');
const httpsAgent = new https.Agent({
rejectUnauthorized: false
});
router.get('/login', (req, res) => {
res.render('login');
});
router.post('/login', (req, res) => {
fetch('https://url.com/api/api_login/', {
method: 'POST',
body: qs.stringify({
'username': req.body.username,
'password': req.body.password
}),
agent: httpsAgent,
credentials: 'include',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
}
})
.then(response => {
console.log(response.headers);
res.cookie("test", "value");
res.render('home');
})
.catch(error => {
console.error(error);
});
Where I'm at so far:
I've fussed around with CORS on both the Django server side and the Express client side (all kinds of combinations of access-control-allow-origin and access-control-allow-credentials). My current iteration of code that I've posted here has it stripped out.
I've fussed around with the cookie settings (httpOnly true/false, secure true/false, SameSite Lax/None, path=/, expires in the future)
I tried axios instead of fetch() and using withCredentials: true
My current iteration is fetch() with credentials: 'include'
The Express console.log(response.headers) actually DOES show the "set-cookie" lines:
Headers {
[Symbol(map)]: [Object: null prototype] {
server: [ 'nginx/1.21.6' ],
date: [ 'Fri, 08 Apr 2022 04:48:49 GMT' ],
'content-type': [ 'text/html; charset=utf-8' ],
'content-length': [ '29' ],
connection: [ 'close' ],
vary: [ 'Accept, Cookie' ],
allow: [ 'POST, OPTIONS' ],
'x-frame-options': [ 'DENY' ],
'x-content-type-options': [ 'nosniff' ],
'referrer-policy': [ 'same-origin' ],
'set-cookie': [
'csrftoken=vNgTkUBruc1xeL27KvBYi9esw12hxK8ohQHWQlur7lmiErddU9FVXRnG0Dxas3v2; expires=Fri, 07 Apr 2023 04:48:49 GMT; Max-Age=31449600; Path=/; SameSite=Lax',
'sessionid=.eJxVjMsOwiAUBf-FtSEgj1KX7v0Gch8gVUOT0q6M_y5NutDtzJzzFhG2tcStpSVOLC5Ci9MvQ6BnqrvgB9T7LGmu6zKh3BN52CZvM6fX9Wj_Dgq00tcjDAGCs8Yo6w0pjQwqe8g2qY6QOeRw1oq7oqyRPA_OOI0q0AjJiM8X2AE34Q:1ncgYD:vRmuQlX4P82-Utw8qmPzSoS-t6Xo7D89CO0UBtyltVY; expires=Fri, 22 Apr 2022 04:48:49 GMT; HttpOnly; Max-Age=1209600; Path=/; SameSite=Lax'
]
}
}
Here's my interpretation, in case it makes obvious what I'm doing wrong. I'm pretty sure the backend is fine - it's supplying the set-cookie header. So it's a frontend issue and a question why the browser isn't consuming that header and setting the cookie. I do get the "test" cookie that I manually set, so I know it's not because my browser is rejecting cookies. I don't think I have a CORS issue because from both the server and client POV, I'm in the same domain (https://url.com), even though the server and client are going to different ports, cookies should be port-agnostic. But just in case, I did try adding CORS headers for "access-control-allow-origin: https://url.com:8003" but that didn't help either. I'm not getting either the csrf cookie or the sessionid cookie.
Postman also does not get the cookie. Postman can get the cookie if it hits the https://url.com/api/api_login/ endpoint directly.

OK, I've come up with a solution, but perhaps wiser people can tell me if this breaks best practice.
Since DRF is providing the set-cookie headers, I'm using "set-cookie-parser" on the frontend to read the header value and set the cookie manually:
const express = require('express');
const router = express.Router();
const qs = require('qs');
const fetch = require('node-fetch');
var setCookie = require('set-cookie-parser');
const api_url = "";
const https = require('https');
const httpsAgent = new https.Agent({
rejectUnauthorized: false
});
router.get('/login', (req, res) => {
res.render('login');
});
router.post('/login', (req, res) => {
fetch(api_url + '/api_login/', {
method: 'POST',
body: qs.stringify({
'username': req.body.username,
'password': req.body.password
}),
agent: httpsAgent,
credentials: 'include',
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
}
})
.then(response => {
const cookies = setCookie.parse(response.headers.raw()['set-cookie'], {
decodeValues: true,
});
cookies.forEach(cookie => {
res.cookie(cookie['name'], cookie['value'], {
expires: cookie['expires'],
httpOnly: cookie['httpOnly'],
maxAge: cookie['maxAge'],
path: cookie['path'],
sameSite: cookie['sameSite'],
secure: cookie['secure'],
})
});
return response.text();
})
.then(text => {
console.log(text);
res.render('home');
})
.catch(error => {
console.error(error);
});
});
module.exports = router;
I'm still not entirely sure why my browser wasn't setting the cookie itself. I started to suspect that while my Express server got the response.header with the "set-cookie" directive, it was not passing it along to my browser. Seeing how my test cookie was being set correctly, I decided just to do it explicitly instead. Is this the correct way? Are there security implications? I have no idea. Because I'm also manually setting all the cookie parameters (httpOnly and Secure), I'm assuming it's "just as safe" as if the browser had consume the set-cookie header and done it itself...

Related

Possible to retrieve a httpOnly cookie with Supertest and send it back in the following request?

I am trying to access an endpoint on my Express server that has an http only cookie as part of the authentication dance to gain access.
Here we set the http only cookie:
res.cookie('jwt', refreshToken, { httpOnly: true, sameSite: 'none', secure: true, maxAge: 3 * 24 * 60 * 60 * 1000 });
return res.send({ accessToken, role: userModel.role, profileId: userModel.profile });
Here is my endpoint I am trying to integration test:
router.delete('/auth/delete/:id', basicAuth, checkAccountId, mentors.deleteMentorProfile);
Inside basicAuth there is this: export const basicAuth = passport.authenticate('jwt', { session: false, failWithError: true });
What I think is required is that the httponly cookie is sent with the request to this endpoint. The line says passport.authenticate('jwt', ...) hence I must need the 'jwt' value to be there in a cookie.
And here is the code from my integration test:
const logInResponse = await api.post('/api/users/authenticate').send(userAcctCreationDetails);
const jwtForTest = logInResponse.body.accessToken;
const allMentors = await Mentor.find({});
const testAccountMentorId = allMentors.find((mentor) => mentor.email === userToDelete.email)?._id.toString();
// act
const cookies = logInResponse.headers['set-cookie'][0];
// console.log(cookies);
const deletedProfileResponse = await api
.delete('/api/mentor/auth/delete/' + testAccountMentorId)
.set('Authorization', 'Bearer ' + jwtForTest)
.set('jwt', cookies); // this fails!
The line console.log(cookies) says some authentication credentials ending in jwt=eyJhbGciOi...-2n21vxTNPOlS94-YeFhSN7o; Max-Age=259200; Path=/; Expires=Thu, 15 Dec 2022 01:26:24 GMT; HttpOnly; Secure; SameSite=None
My coworker tells me "httponly means the client can't access it" but can I at least send it back? How does a browser send it back if not via javascript?
Note I did google it and i found extreme scarcity of information about supertest and HTTP only cookies.

fetch() Not setting cookie despite `Set-Cookie` header

My backend-api written in Node and Express.js sets a cookie using res.cookie:
Router.post('/login', async (req, res) => {
const email = req.body.email;
const password = req.body.password;
try {
let result = await sqlite.login(email, password);
res.cookie('token', result, {
'maxAge': 3600 * 1000
});
res.send({
'token' : result
});
} catch (err) {
res.send(err);
}
});
I can make a request to this route, and I do notice the Set-Cookie header is set on the response object within Chrome developer tools:
Set-Cookie: token=[...]; Max-Age=3600; Path=/; Expires=Mon, 11 Jul 2022 14:47:08 GMT
However, document.cookie is never set by the browser. From my searching, most people say to specify the credentials field as same-origin. I have done this and it made no change. My cookie is NOT being set as HttpOnly, so I am unsure why it's being set by the browser.
Here is where I call the /login route:
async login(email, password) {
let response = await fetch(apiURL + '/login', {
'method' : 'POST',
'headers' : {
'Content-Type' : 'application/json',
'Accept' : 'application/json'
},
'credentials' : 'same-origin',
'body' : JSON.stringify({
'email' : email,
'password' : password
})
});
return await response.json();
}
A token is successfully returned in the response, but again document.cookie returns an empty string ''.
From searching this problem, most of the issues seem to suggest that same-origin should fix the issue but it is not the case for myself. Another thing of note is that httpOnly cookies won't show in the browser, but I know that the cookies I am sending are not HttpOnly.
I am using Google Chrome version 103.0.5060.114.
If I set credentials to include, I get a CORS error:
The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'.
I was able to get it working with the following changes:
When initializing npm package cors, specify some options:
app.use(cors({ credentials: true, origin: 'http://lvh.me:3001' }));
The origin must include the http:// prefix as well as the correct port. Once that was done, set credentials to include when calling fetch() and it should work fine.

httponly cookies not set

Having this setup:
auth server (keycloak): localhost:9990,
backend: localhost:8080,
frontend (SPA): localhost:3000
Users can "login" / obtain a token on the SPA by sending their username + password to:
http://localhost:9990/auth/realms/w/protocol/openid-connect/token
Then I call another url on the auth-server that should set the cookies that keycloak needs for SSO/remember-me (it should set some HttpOnly cookies):
.then(t => keycloakInstance.init({
token: t.access_token,
refreshToken: t.refresh_token,
checkLoginIframe: false, // required to init with token
})
.then((authenticated) => {
console.log('auth', authenticated); // <-- it is true
if (authenticated) {
return fetch('http://localhost:9990/auth/realms/w/custom-sso-provider/sso', { headers: {
Authorization: `Bearer ${keycloakInstance.token}` } })
// else
The request itself seems fine, the Set-Cookie occurs as I would expect; this is the response header:
I would now expect them to occur in devtools > Application > cookies, but unfortunately no cookies show up. Why? And what can I do about it?
I was missing credentials: 'include' for the fetch call:
return fetch('http://localhost:9990/auth/realms/w/custom-sso-provider/sso', { headers: { Authorization: `Bearer ${keycloakInstance.token}` }, credentials: 'include' })

Apify: Preserve headers in RequestQueue

I'm trying to crawl our local Confluence installation with the PuppeteerCrawler. My strategy is to login first, then extracting the session cookies and using them in the header of the start url. The code is as follows:
First, I login 'by foot' to extract the relevant credentials:
const Apify = require("apify");
const browser = await Apify.launchPuppeteer({sloMo: 500});
const page = await browser.newPage();
await page.goto('https://mycompany/confluence/login.action');
await page.focus('input#os_username');
await page.keyboard.type('myusername');
await page.focus('input#os_password');
await page.keyboard.type('mypasswd');
await page.keyboard.press('Enter');
await page.waitForNavigation();
// Get cookies and close the login session
const cookies = await page.cookies();
browser.close();
const cookie_jsession = cookies.filter( cookie => {
return cookie.name === "JSESSIONID"
})[0];
const cookie_crowdtoken = cookies.filter( cookie => {
return cookie.name === "crowd.token_key"
})[0];
Then I'm building up the crawler structure with the prepared request header:
const startURL = {
url: 'https://mycompany/confluence/index.action',
method: 'GET',
headers:
{
Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7',
Cookie: `${cookie_jsession.name}=${cookie_jsession.value}; ${cookie_crowdtoken.name}=${cookie_crowdtoken.value}`,
}
}
const requestQueue = await Apify.openRequestQueue();
await requestQueue.addRequest(new Apify.Request(startURL));
const pseudoUrls = [ new Apify.PseudoUrl('https://mycompany/confluence/[.*]')];
const crawler = new Apify.PuppeteerCrawler({
launchPuppeteerOptions: {headless: false, sloMo: 500 },
requestQueue,
handlePageFunction: async ({ request, page }) => {
const title = await page.title();
console.log(`Title of ${request.url}: ${title}`);
console.log(page.content());
await Apify.utils.enqueueLinks({
page,
selector: 'a:not(.like-button)',
pseudoUrls,
requestQueue
});
},
maxRequestsPerCrawl: 3,
maxConcurrency: 10,
});
await crawler.run();
The by-foot-login and cookie extraction seems to be ok (the "curlified" request works perfectly), but Confluence doesn't accept the login via puppeteer / headless chromium. It seems like the headers are getting lost somehow..
What am I doing wrong?
Without first going into the details of why the headers don't work, I would suggest defining a custom gotoFunction in the PuppeteerCrawler options, such as:
{
// ...
gotoFunction: async ({ request, page }) => {
await page.setCookie(...cookies); // From page.cookies() earlier.
return page.goto(request.url, { timeout: 60000 })
}
}
This way, you don't need to do the parsing and the cookies will automatically be injected into the browser before each page load.
As a note, modifying default request headers when using a headless browser is not a good practice, because it may lead to blocking on some sites that match received headers against a list of known browser fingerprints.
Update:
The below section is no longer relevant, because you can now use the Request class to override headers as expected.
The headers problem is a complex one involving request interception in Puppeteer. Here's the related GitHub issue in Apify SDK. Unfortunately, the method of overriding headers via a Request object currently does not work in PuppeteerCrawler, so that's why you were unsuccessful.

Next.js not persisting cookies

I have a server-side rendered Next.js/express app that communicates with a Django API (cross-origin). I login a user like so:
const response = await fetch('localhost:8000/sign-in', {
method: 'POST',
credentials: 'include',
body: JSON.stringify({ email, password }),
headers: { 'Content-Type': 'application/json' },
});
const result = await response.json();
if (response.status === 200) {
Router.push('/account');
}
Django successfully logs in the user and returns set-cookie headers for the csrftoken and sessionid cookies, however, when I navigate to a different page (like in the above code when I Router.push), the cookies don't persist.
I assume this has something to do with server-side vs. client-side, but when cookies are set in the browser I expect them to persist regardless.
How can I get these cookies, once set, to persist across all pages on the client side?
It turns out that set-cookie is the old way of doing things. It's controlled by the browser, so it's obfuscated.
I ended up sending the csrftoken and sessionid back to the client in the JSON body, and saving them to localStorage using localStorage.setItem('sessionid', 'theSessionId') and localStorage.setItem('csrftoken', 'theCsrftoken').
Then when I need to make an authenticated request, I include them in the fetch headers:
const response = await fetch(`${API_HOST}/logout`, {
method: 'POST',
headers: {
'X-CSRFToken': localStorage.getItem('csrftoken'),
sessionid: localStorage.getItem('sessionid'),
},
});