I'm learning how to use django sessions and caching.
I noticed that once I login, a sessionId is included in the cookie header. I'm using the "#authentication_classes" and "#permission_classes" decorators to validate the session.
I'm also using Redis to cache the session, since I dont want to hit the DB everytime. I followed the docs for the django and redis server setup.
Once the user log in, the session is written in the cache. The problem is that, even tho the session is inside the cache, the database is hit with each request and it slows down the response time to around 300ms.
This is the api I'm using for testing. It does nothing, but it takes 300 ms and it does hit the DB each time I call it. I'm using "never_cache" because I dont want the entire response to be cached.
#api_view(["GET"])
#authentication_classes ([SessionAuthentication])
#permission_classes([IsAuthenticated])
#never_cache
def test(request, username):
return JsonResponse("", safe=False)
this is the request I'm making:
request
If I remove the session autentication decorators, a SessionId is not required anymore to access the api, but the DB is still hit and the response still takes 300ms (there is still the sessionID token inside the cookie header).
#api_view(\["GET"\])
#never_cache
def test(request, username):
return JsonResponse("", safe=False)
Now, if I remove the sessionId from the cookie header the DB is not hit anymore and the response only takes 20ms.
request2
I also tried creating a new header called "sessionId" and passing the sessionId, the request is still fast, but the django decorators are not able anymore to validate the session.
This is the key in the redis server:
redis
Those are my middleware and cache settings in the setting.py file:
settings
I really don't understand what I'm doing wrong.
Any tips / suggestion is welcomed.
Thank you for your time and patience.
Related
I've been scratching my head for the last couple days, trying to figure out an appropriate fix for this issue.
I'm using Flask-Login with Flask-WTF. Upon logging in, I am able to submit all POST requests to CSRF protected forms through Flask-WTF, and everything works as expected.
However, if "Remember_Me" has been set (thus placing a remember me cookie on the client side) and a close and restart my browser, on visiting the form two things happen:
1 - Flask Login captures from the remember me cookie, correctly logging
in and authenticating.
2 - Attempmting to POST any data, either Ajax or
through a form, Flask-WTF fails CSRF validation, due to line 86 in
Flask-WTF/csrf.py
if field_name not in session:
raise ValidationError('The CSRF session token is missing.')
This is I'm guessing expected behaviour, as the session token doesn't theoretically exist for a remembered session, as Flask-Login pulls the user ID out of the remember_me cookie.
Anyone come across this issue at all? I have no idea how to go about sorting this - I don't want to disable CSRF protection, nor allowing remember_me tokens.
Is there some way I can refresh the session token on successful retrieval from the cookie?
I want to crawl a website which has a strong security protocol and want to crawl data as fast as possible. Thus I thought I need a multi-login-cookie, multi-user-agent, and multi-proxy crawler.
I have tens of usernames and passwords and I can login using each one and get all the cookies. To hide the identity of my crawler I thought I should also replace the user-agent setting and my IP. I have found many user agents and proxies.
I learned that the cookie is needed each time I send a request to the server, and that cookie should be of the same identity and contain the information of the previous request and the corresponding response. I've gained the knowledge of how to pass it through requests without logging in from this answer. And I know two ways to login in, one outside the scrapy(by passing the cookie to the cookiesmiddleware in the middleware.py file:
from cookies import cookies # script written to login some accounts and return the cookies
import random
class CookiesMiddleware(object):
def process_request(self, request, spider):
cookie = random.choice(cookies)
request.cookies = cookie
) and another inside it.
What's more in the middleware.py file I passed the user agents randomly in the same as for cookies to the scrapy requests.
My question is: if I pass the cookies randomly as aforementioned, will one spider get the same cookie each time it sends a request? If not the server side will detect me as a bot and block me. What's worse, the same applies to the user-agents and proxies. How to bond each trinity(login cookie, user-agent and proxy) starting from the login, extending the aforesaid answer both in the horizontal and vertical dimension?
To be more precise, should I pass the login cookie in the form of {cookies= user1_cookie} or { meta={'cookiejar': user1_cookie},? And should I pass the user agent and proxy in the meta parameter?
Thanks. Please kindly point me in the right direction, and any suggestions will be highly received and appreciated.
Seems like you are looking for cookiejar. It will allow you to store multiple cookie sessions in single spider session.
Using middleware for random cookies is a bad idea since cookies in most cases store your whole browsing sessions.
We are currently developing a mobile application(using ionic) for our site that was built using Django. We used django sessions on the site everytime a user logs in. From what I understand, django session sets the session id in the client that is stored in the cookie of the browser. How can we set this session id in the mobile app, if the mobile app is separate from django?
I see at least three ways you can approach this:
find a way to get your ionic app to work with django's cookies (session and CSRF).
change your django app in three ways: 1) have login/logout views that give an authentication token to the mobile application, 2) extend SessionMiddleware to extract the session id from the authentication token sent via the "Authorization" HTTP header of the mobile app's requests, 3) extend django's CSRF middleware to make requests that do not come from web browsers exempt from CSRF checks.
try to use existing solutions for adding JWT (Json Web Token) authentication to Django.
Getting WebView-based hybrid mobile apps to work properly with cookies on both android and ios seems to be non-trivial (e.g. Handling cookies in PhoneGap/Cordova).
The existing solutions for JWT-auth I found thus far don't seem to use Django's sessions (which I find rather convenient for the sake of having a uniform solution for killing the sessions of users who get their phones/computers stolen or their accounts hacked). If someone knows of a pluggable solution that does token-authentication on top of django's regular sessions, please comment.
To elaborate on method 2:
When your mobile app doesn't handle cookies, it can't even log in due to Django's CSRF check. That's why requests coming from your mobile app need to be exempt from CSRF protection.
Note: Do NOT simply disable CSRF protection. While the mobile application isn't vulnerable to CSRF attacks, the browsers of people who are visiting the existing website still are.
To extend the CSRF mechanism to exempt non-browser requests, you need an effective way to determine whether a request is coming from a web browser. There's a bunch of HTTP headers (E.g. Referer, Cookie, X-Requested-With, Origin) we can look at to get an idea. Let's assume for now that your legitimate website users don't spoof their headers.
Use the same "is-web-browser"-check you used for CSRF-exemption to protect the mobile app's login/logout views.
Example Code (for an Android mobile app, iOS headers coming from the mobile app are likely different):
def is_mobile_app_access(request):
return request.META.get('HTTP_REFERER', None) is None
and request.META.get('HTTP_COOKIE', None) is None
and request.META.get('HTTP_X_REQUESTED_WITH', None) == 'your.app.name.here'
and request.META.get('HTTP_ORIGIN', None) == 'file://'
class CustomCsrfViewMiddleware(CsrfViewMiddleware):
def process_view(self, request, callback, callback_args, callback_kwargs):
if is_mobile_app_access(request):
return None
else:
return super(CustomCsrfViewMiddleware, self).process_view(request, callback, callback_args, callback_kwargs)
def process_response(self, request, response):
if is_mobile_app_access(request):
return response
else:
return super(CustomCsrfViewMiddleware, self).process_response(request, response)
class CustomSessionMiddleware(SessionMiddleware):
def process_request(self, request):
if is_mobile_app_access(request):
session_key = request.META.get("HTTP_AUTHORIZATION", None)
else:
session_key = request.COOKIES.get(settings.SESSION_COOKIE_NAME)
request.session = self.SessionStore(session_key)
If anyone sees something fundamentally wrong with this approach, please let me know. I suppose the weakest link in this approach is how reliable the is_mobile_app_access check is.
What did you end up doing?
After my site has been up and running for a while in production, I suddenly have a problem with my users loging into it.
I have protected certain views/pages with the login_required decorator and I am also using the django admin.
When an anonymous user hits any of these pages, he is redirected to the login page.
When this anonymous user adds its credentials, the POST request is successful and he is redirected to the inital page. At the same time, the user gets a new sessionid (as expected)
However, now the results get very unreliable. When pressing reload or when navigating to other pages (that require a login), either of the 2 outcomes might happen:
a) The user is identified and the page is displayed correctly
b) The user is redirect to the login page.
I have checked the content of the session via the shell, and nothing is changing there.
The production site is served via a load balancer and 8 application servers. Even stranger: if I test the same code (with the same settings) on a test server, that is not load balanced and basically has not traffic, everything is working fine.
I am running Django 1.6 on Ubuntu with Apache and mod_wsgi in daemon mode behind SSL and I am using the Session database backend. I am using django-allauth.account for account management/login. My Session settings are like this:
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_AGE = 60*60*24
SESSION_COOKIE_SECURE = True
UPDATE
To get some more debug information, I have created this Middleware:
from django.conf import settings
class SessionDebugMiddleware(object):
def process_response(self, request, response):
session = request.session
user = getattr(request, 'user', None)
if user:
user=user.id
session_key = request.COOKIES.get(settings.SESSION_COOKIE_NAME, None)
response['X-Meta-Requ'] = '{0},{1},{2},{3}'.format(session_key, session.get('_auth_user_id'), session.get('_auth_user_backend','---'), user)
return response
If I hit the refresh button 10 times,
8 times I will get this header:
igv0xshezhdxh50kks9x00r0l67mx0sk,None,---,None
2 times I will get this header: X-Meta-Requ:igv0xshezhdxh50kks9x00r0l67mx0sk,330619,django.contrib.auth.backends.ModelBackend,330619
It seems be be random and does not follow any logic.
So I have the following questions/ideas?
Can this be related to loadbalancing in anyway? My understanding was that Django does not need sticky sessions, when the DB session backend is used.
Can this be related to a threading issue?
Can this be related to high load?
Can this be related to a decoding issue: https://github.com/django/django/blob/master/django/contrib/sessions/backends/base.py#L83. But why should that decoding issue not be consistent. And I have not found any log entries that refer to "Session data corrupted".
Any other hints are welcome.
Just in case someone is having this issue, it can also be caused by SECRET_KEY definitions not being consistent.
I had mine generated in the settings file (a bad decision).
It was fine in development mode, but once the application was served by WSGI, multiple instances were created, each with different keys.
This causes the mixup in the hash, and the corrupt session to be reported.
I think you have problem because of highload or error in your code (maybe you just rebuild session in your application anywhere and forget about it :)
Please try to read info from database with your session key to be sure session data exists.
If data exists but not loaded to session - this mean you have troubles with session processing. If data doesn't exists, but it present in past because you got 2 times right session data, this mean you have problems with session storage (maybe because of highload your database lose sessions data). Do you have database replication?
Also which session backend you use? Just a DB or Cached_Db?
We recently changed our cookie domain (settings.SESSION_COOKIE_DOMAIN) from domain.com to .domain.com and this is now preventing Safari users from logging in unless they clear their cookies. The problem seeds to the fact that there's sessionid cookies set for both domains.
Is there a way to clear or ignore the original domain or is the only way to advice users to clear cookies?
This can occur when the SESSION_COOKIE_DOMAIN setting is modified. (You stated that SESSION_COOKE_PATH was changed).
The documentation for SESSION_COOKIE_DOMAIN comes with this warning:
SESSION_COOKIE_DOMAIN
Be cautious when updating this setting on a production site. If you
update this setting to enable cross-domain cookies on a site that
previously used standard domain cookies, existing user cookies will be
set to the old domain. This may result in them being unable to log in
as long as these cookies persist.
This would occur if you went from:
SESSION_COOKIE_DOMAIN = None
to
SESSION_COOKIE_DOMAIN = '.domain.com'
As you've stated, there are now two cookies on the client end which will be sent to the server during a request, with both cookies named sessionid. When Django looks at the cookies, it only has access to a Python dictionary, so it only every sees one sessionid cookie rather than both that were sent.
I haven't tested this, but some ideas for fixing the problem may be:
Ask the user to delete the appropriate cookies. Depending on the number of users and their skill level, this may not be an reasonable option. Asking them to delete ALL their cookies may be out of the question.
Wait for the old cookies to expire. By default, it appears as though the sessionid cookie has a 14 day expiry. Once the old session cookies expire, they will no longer be sent with each request, allowing the new sessionid cookie to take effect.
Change the name of the sessionid cookie and write a custom Django middleware to handle both old and new sessionid cookies.
I haven't tested the last point, but it should be possible to change the SESSION_COOKIE_NAME to something other than sessionid. Now, this will prevent existing logged in users from using their existing sessionid cookies, so you'd need to write a custom middleware which was capable of handling both sessionid cookies (for old cookies) and sessionidnew cookies for current logons.
Something like this will work:
from django.utils.importlib import import_module
from django.contrib.sessions import middleware
from django.conf import settings
class MySessionMiddleware(middleware.SessionMiddleware):
def process_request(self, request):
engine = import_module(settings.SESSION_ENGINE)
session_key = request.COOKIES.get(settings.SESSION_COOKIE_NAME, None)
if session_key is None:
# Look for old cookie in request for auth purposes.
session_key = request.COOKIES.get('sessionid', None)
request.session = engine.SessionStore(session_key)
You'd have to replace the SessionMiddleware in settings.py under MIDDLEWARE_CLASSES with this new middleware. Eg: Change 'django.contrib.sessions.middleware.SessionMiddleware' to 'custommiddleware.MySessionMiddleware' where custommiddleware.py is the file with above code and is present in your project root folder (where manage.py file is present)
Once enough time has passed and you're satisfied that all of the old sessionid cookies have expired, you could do the reverse and change back to using sessionid as the preferred cookie name for sessions, eventually removing the specialist code which can handle two different types of sessionid cookie.