I'm using code below to debug an issue where it doesn't appear my session variable is being updated properly.
print(session)
session['review_status'] = 'Pending'
print('Session review_status is now: ' + session['review_status'])
print(session)
This is outputting the following:
<SecureCookieSession {'review_id': None, 'review_status': 'New'}>
Session review_status is now: Pending
<SecureCookieSession {'review_id': None, 'review_status': 'New'}>
I can't understand why the last print statement isn't reflecting that the review_status value should now be "Pending" and not "New".
The frontend is firing off about 5 ajax requests at once to this endpoint, but the first one should be changing the status to Pending, so by the time the other 4 return, it would be "Pending" for them.
It appears this was being caused by the asynchronous calls from the frontend. While watching the debug output, it seems that several of the calls were "finishing" before the Flask session was actually reflecting the new value that was stored.
When I converted the ajax call to "async: false", I then got the expected behavior after the first call finished (the remainder were no longer "New", but rather in "Pending").
I am going to leave this fix in place for now, but would be interested in alternatives to this, and getting a better understanding of how Gunicorn/Flask handles multiple requests to the same endpoint concurrently with regard to the session (ie: does the session remain static until all calls are fulfilled type thing).
Related
This makes the vary_on_cookie decorator useless because the coookie is different at every request, and also it makes the sessions never expire regardless of SESSION_COOKIE_AGE because every request pushes the expiration forward.
I did some digging, and process_response in sessions/backends/middleware.py does request.session.save(), which causes the signed_cookie backend to create a new session key with this code:
return signing.dumps(
self._session, compress=True,
salt='django.contrib.sessions.backends.signed_cookies',
serializer=self.serializer,
)
This code returns a different value even if called with the same argument (I guess it's built into the encryption).
Am I missing something? This doesn't sound right... any ideas?
Thanks!
Figured it out: one of my middlewares was modifying the session in process_request, which caused the session object to be modified, and therefore the chain of events explained in the question happened.
I'm building an ember app, and I keep running into the same problem where I make a call to the store, but the function keeps compiling before the store has retrieved the data from the backend. Specifically I'm having the problem with a findRecord. I've implemented it both ways:
var admin = this.store.findRecord('admin', 1);
console.log(admin.get('season'));
console.log('Are we here?');
and
this.store.findRecord('admin', 1).then(function(admin) {
console.log(admin.get('season'));
});
console.log('Are we here?');
In both cases, the Are we here? is logged BEFORE the season. Obviously the console logs are just for the example, and it creates an actual problem with what I'm trying to get done. Does anybody know a simple fix for this delay?
Thanks.
Of course it is. It's an asynchronous behavior. It takes some time to solve promise which is returned from findRecord(), thus the consequence is:
this.store.findRecord(); //synchronous
console.log('we are here'); //synchronous
in the meantime a promise returned from findRecord() gets resolved (asynchronous behavior)
console.log(admin.get('season'));
An asynchronous call will not stop your code from progressing, that´s the purpose of it. Else it would block UI updates and user interaction while loading data.
I'm trying to cache a set of strings per session by storing each one in their own variable and by using django.contrib.session.
I have the following code:
import copy
def get_result(request, operation):
previous_result = request.session.get(operation.name)
if previous_result:
result = copy.deepcopy(previous_result)
else:
result = get_json_response(operation)
request.session[operation.name] = copy.deepcopy(result)
return result
get_result() is
triggered via ajax requests
used for many different operations which may be called at the same time
may be called multiple times per operation in one session
This code works perfectly fine on my local environment. However, in production server where gevent and chausette is installed, it fails.
Most of the time, request.session.get(operation.name) would return None even when it is not the first time that get_result is called for that operation. In some cases, it returns a value but in some, it doesn't. There seems to be no pattern on when it does and doesn't work.
I suspect that the inconsistency is because different threads are referencing the session variable at different states. What would be the proper way to handle session variables in this case?
I did in fact have the same problems and also tried to save the session properly with the tweaks you posted.
In the end, what solved my problem was changing the default cache in settings.py to
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
Using FileBasedCache instead helps as well, but it crashes in the local environment (development). Dummy works for local as well as production.
We developed a REST API using Django & mongoDB (PyMongo driver). The problem is that, on some requests to the API endpoints, PyMongo cursor returns a partial response which contains less documents than it should (but it’s a completely valid JSON document).
Let me explain it with an example of one of our views:
def get_data(key):
return collection.find({'key': key}, limit=24)
def my_view(request):
key = request.POST.get('key')
query = get_data(key)
res = [app for app in query]
return JsonResponse({'list': res})
We're sure that there is more than 8000 documents matching the query, but in
some calls we get less than 24 results (even zero). The first problem we've
investigated was that we had more than one MongoClient definition in our code. By resolving this, the number of occurrences of the problem decreased, but we still had it in a lot of calls.
After all of these investigations, we've designed a test in which we made 16 asynchronous requests at the same time to the server. With this approach, we could reproduce the problem. On each of these 16 requests, 6-8 of them had partial results. After running this test we reduced uWsgi’s number of processes to 6 and restarted the server. All results were good but after applying another heavy load on the server, the problem began again. At this point, we restarted uwsgi service and again everything was OK. With this last experiment we have a clue now that when the uwsgi service starts running, everything is working correctly but after a period of time and heavy load, the server begins to return partial or empty results again.
The latest investigation we had was to run the API using python manage.py with DEBUG=False, and we had the problem again after a period of time in this situation.
We can't figure out what the problem is and how to solve it. One reason that we can think of is that Django closes pymongo’s connections before completion. Because the returned result is a valid JSON.
Our stack is:
nginx (with no cache enabled)
uWsgi
MemCached (disabled during debugging procedure)
Django (v1.8 on python 3)
PyMongo (v3.0.3)
Your help is really appreciated.
Update:
Mongo version:
db version v3.0.7
git version: 6ce7cbe8c6b899552dadd907604559806aa2e9bd
We are running single mongod instance. No sharding/replicating.
We are creating connection using this snippet:
con = MongoClient('localhost', 27017)
Update 2
Subject thread in Pymongo issue tracker.
Pymongo cursors are not thread safe elements. So using them like what I did in a multi-threaded environment will cause what I've described in question. On the other hand Python's list operations are mostly thread safe, and changing snippet like this will solve the problem:
def get_data(key):
return list(collection.find({'key': key}, limit=24))
def my_view(request):
key = request.POST.get('key')
query = get_data(key)
res = [app for app in query]
return JsonResponse({'list': res})
My very speculative guess is that you are reusing a cursor somewhere in your code. Make sure you are initializing your collection within the view stack itself, and not outside of it.
For example, as written, if you are doing something like:
import ...
import con
collection = con.documents
# blah blah code
def my_view(request):
key = request.POST.get('key')
query = collection.find({'key': key}, limit=24)
res = [app for app in query]
return JsonResponse({'list': res})
You could end us reusing a cursor. Better to do something like
import ...
import con
# blah blah code
def my_view(request):
collection = con.documents
key = request.POST.get('key')
query = collection.find({'key': key}, limit=24)
res = [app for app in query]
return JsonResponse({'list': res})
EDIT at asker's request for clarification:
The reason you need to define the collection within the view stack and not when the file loads is that the collection variable has a cursor, which is basically how the database and your application talk to each other. Cursors do things like keep track of where you are in a long list of data, in addition to a bunch of other stuff, but thats the important part.
When you create the collection cursor outside the view method, it will re-use the cursor on each request if it exists. So, if you make one request, and then another really, really fast right after that (like what happened when you applied high load), the cursor might only be half way through talking to the database, and so some of your data goes to the first request, and some to the second. The reason you would get NO data in a request would be if a cursor finished fetching data but hadn't been closed yet, so the next request tried to fetch data from the cursor, and there was none left to fetch in the query.
By moving the collection definition (and by association, the cursor definition) into the view stack, you will ALWAYS get a new cursor when you process a new request. You wont get any cross talking between your cursors and different requests, as each request cycle will have its own.
EDIT: This question fixed my issue: What happens when no response is received for a request? I'm seeing retries
I have a weird issue. I have a Django application. The frontend has two forms. I submit the forms together via the same Ajax post method
// code to get and format form data to get into the format I need in the backend
var webURL = "showFormOutput";
$.post(webURL,
dataToSend,
callback
).fail(function(jqXHR, textStatus, errorThrown)
{
console.log("Status: " + textStatus);
console.log("Error: " + errorThrown);
});
In the Django view corresponding to that url, I do the following:
Print a message "View function" to log that I am in the view.
Parse the post data and put it into appropriate variables. Pass these variables to a Python function in a separate file. Let us call this function getQueryResults.
Create two Postgres queries (for two forms) using these variables. Print a log message "Query created".
Use Python's threading module to execute these two queries. I run the following line twice to create two separate threads for the two queries threading.Thread(target=threadFunction, args=(query,).start()
Log in the threadFunction "Query result ready" when the postgres query returns.
This is a long running query - takes up to ten minutes.
In Chrome, this works perfectly. However, in Firefox (and Safari too I think), I get the following issues:
When using Firefox, I can see in pg_stat_activity that two queries are running as expected. However, after about 5 minutes, there are FOUR queries running i.e. two more of the SAME query are started.
This is reflected in my logs also. I get the messages "View Function" and "Query Created" printed AGAIN. This means that somehow, the execution restarted from the view. This happens even on increasing the http.response.timeout to 30000.
Sometimes, Firefox just goes to the fail() case of the $.post and textStatus just says "Error" and errorThrown is blank. Sometimes, it prints my query results as expected but it waits for the second set of queries to finish. I thought it might be an issue with my Python threads but it makes no sense to have the whole view executed again since the threads never call the view anywhere! On the client side, the POST request doesn't seem to be sent again.
Notes: I am using django's filesystem-based cache to cache the query results. Could this be an issue? The second duplicate set of queries turn up even before the first actual set return results so I doubt this is the issue though.
Has anyone got any idea why using Firefox would result in my view being called twice?