EDIT: This question fixed my issue: What happens when no response is received for a request? I'm seeing retries
I have a weird issue. I have a Django application. The frontend has two forms. I submit the forms together via the same Ajax post method
// code to get and format form data to get into the format I need in the backend
var webURL = "showFormOutput";
$.post(webURL,
dataToSend,
callback
).fail(function(jqXHR, textStatus, errorThrown)
{
console.log("Status: " + textStatus);
console.log("Error: " + errorThrown);
});
In the Django view corresponding to that url, I do the following:
Print a message "View function" to log that I am in the view.
Parse the post data and put it into appropriate variables. Pass these variables to a Python function in a separate file. Let us call this function getQueryResults.
Create two Postgres queries (for two forms) using these variables. Print a log message "Query created".
Use Python's threading module to execute these two queries. I run the following line twice to create two separate threads for the two queries threading.Thread(target=threadFunction, args=(query,).start()
Log in the threadFunction "Query result ready" when the postgres query returns.
This is a long running query - takes up to ten minutes.
In Chrome, this works perfectly. However, in Firefox (and Safari too I think), I get the following issues:
When using Firefox, I can see in pg_stat_activity that two queries are running as expected. However, after about 5 minutes, there are FOUR queries running i.e. two more of the SAME query are started.
This is reflected in my logs also. I get the messages "View Function" and "Query Created" printed AGAIN. This means that somehow, the execution restarted from the view. This happens even on increasing the http.response.timeout to 30000.
Sometimes, Firefox just goes to the fail() case of the $.post and textStatus just says "Error" and errorThrown is blank. Sometimes, it prints my query results as expected but it waits for the second set of queries to finish. I thought it might be an issue with my Python threads but it makes no sense to have the whole view executed again since the threads never call the view anywhere! On the client side, the POST request doesn't seem to be sent again.
Notes: I am using django's filesystem-based cache to cache the query results. Could this be an issue? The second duplicate set of queries turn up even before the first actual set return results so I doubt this is the issue though.
Has anyone got any idea why using Firefox would result in my view being called twice?
Related
I have a Django application where I am retrieving information from remote Oracle database and I populate those information in chart using Chart.js.
As you can see in my code, I am calling two times connection to database since I am populating two charts
The point is that process of retrieving information sometimes is taking so much time, and I read there is option to include async await option , where I can decrease I/O time.
(attach and gngp_ntp is query that I am taking from other .py file)
Can someone help me how to make simple async below function.
def index(request):
cursor1=connections['default'].cursor()
cursor1.execute(attach) // execute the query
r1= dictfetchall(cursor1)
cursor2=connections['default'].cursor()
cursor2.execute(gngp_ntp)// execute the query
r2= dictfetchall(cursor2)
return render(request,'app_sins/ps_capacity.html', {"all_post" : r1,"jokso" : r2})
I'm using code below to debug an issue where it doesn't appear my session variable is being updated properly.
print(session)
session['review_status'] = 'Pending'
print('Session review_status is now: ' + session['review_status'])
print(session)
This is outputting the following:
<SecureCookieSession {'review_id': None, 'review_status': 'New'}>
Session review_status is now: Pending
<SecureCookieSession {'review_id': None, 'review_status': 'New'}>
I can't understand why the last print statement isn't reflecting that the review_status value should now be "Pending" and not "New".
The frontend is firing off about 5 ajax requests at once to this endpoint, but the first one should be changing the status to Pending, so by the time the other 4 return, it would be "Pending" for them.
It appears this was being caused by the asynchronous calls from the frontend. While watching the debug output, it seems that several of the calls were "finishing" before the Flask session was actually reflecting the new value that was stored.
When I converted the ajax call to "async: false", I then got the expected behavior after the first call finished (the remainder were no longer "New", but rather in "Pending").
I am going to leave this fix in place for now, but would be interested in alternatives to this, and getting a better understanding of how Gunicorn/Flask handles multiple requests to the same endpoint concurrently with regard to the session (ie: does the session remain static until all calls are fulfilled type thing).
My backend always responds with all available data and it took a considerably amount of time. So I'm reloading store periodically and I plan to use peekAll() and peekRecord().
My code is:
model: function() {
return Ember.RSVP.hash({
'clusters': this.store.peekAll('cluster'),
'single': this.store.peekRecord('cluster', 'cluster::My')
});
When code is executed, at first I can see that both of these items do not contain content. After few seconds data are loaded to store and I can see content 'clusters' on template as expected. But 'single' is still completely without content ({{model.single}} does not return nothing in template). But when I have a button with action:
alert(this.store.peekRecord('cluster', 'cluster::My'));
I can see that the record was found. Records are also available via Ember Inspector. What am I doing wrong that only peekAll() works in model for me.
The semantics of both methods are:
store.peekAll returns a live array that is updated as the store is updated.
store.peekRecord returns the corresponding object in the current cache, or null, and it does not update.
So the behaviour you're observing is the expected one. If you want to use the peek methods, my advise is to make sure that the initial request has finished loading before fetching any data from the store.
We developed a REST API using Django & mongoDB (PyMongo driver). The problem is that, on some requests to the API endpoints, PyMongo cursor returns a partial response which contains less documents than it should (but it’s a completely valid JSON document).
Let me explain it with an example of one of our views:
def get_data(key):
return collection.find({'key': key}, limit=24)
def my_view(request):
key = request.POST.get('key')
query = get_data(key)
res = [app for app in query]
return JsonResponse({'list': res})
We're sure that there is more than 8000 documents matching the query, but in
some calls we get less than 24 results (even zero). The first problem we've
investigated was that we had more than one MongoClient definition in our code. By resolving this, the number of occurrences of the problem decreased, but we still had it in a lot of calls.
After all of these investigations, we've designed a test in which we made 16 asynchronous requests at the same time to the server. With this approach, we could reproduce the problem. On each of these 16 requests, 6-8 of them had partial results. After running this test we reduced uWsgi’s number of processes to 6 and restarted the server. All results were good but after applying another heavy load on the server, the problem began again. At this point, we restarted uwsgi service and again everything was OK. With this last experiment we have a clue now that when the uwsgi service starts running, everything is working correctly but after a period of time and heavy load, the server begins to return partial or empty results again.
The latest investigation we had was to run the API using python manage.py with DEBUG=False, and we had the problem again after a period of time in this situation.
We can't figure out what the problem is and how to solve it. One reason that we can think of is that Django closes pymongo’s connections before completion. Because the returned result is a valid JSON.
Our stack is:
nginx (with no cache enabled)
uWsgi
MemCached (disabled during debugging procedure)
Django (v1.8 on python 3)
PyMongo (v3.0.3)
Your help is really appreciated.
Update:
Mongo version:
db version v3.0.7
git version: 6ce7cbe8c6b899552dadd907604559806aa2e9bd
We are running single mongod instance. No sharding/replicating.
We are creating connection using this snippet:
con = MongoClient('localhost', 27017)
Update 2
Subject thread in Pymongo issue tracker.
Pymongo cursors are not thread safe elements. So using them like what I did in a multi-threaded environment will cause what I've described in question. On the other hand Python's list operations are mostly thread safe, and changing snippet like this will solve the problem:
def get_data(key):
return list(collection.find({'key': key}, limit=24))
def my_view(request):
key = request.POST.get('key')
query = get_data(key)
res = [app for app in query]
return JsonResponse({'list': res})
My very speculative guess is that you are reusing a cursor somewhere in your code. Make sure you are initializing your collection within the view stack itself, and not outside of it.
For example, as written, if you are doing something like:
import ...
import con
collection = con.documents
# blah blah code
def my_view(request):
key = request.POST.get('key')
query = collection.find({'key': key}, limit=24)
res = [app for app in query]
return JsonResponse({'list': res})
You could end us reusing a cursor. Better to do something like
import ...
import con
# blah blah code
def my_view(request):
collection = con.documents
key = request.POST.get('key')
query = collection.find({'key': key}, limit=24)
res = [app for app in query]
return JsonResponse({'list': res})
EDIT at asker's request for clarification:
The reason you need to define the collection within the view stack and not when the file loads is that the collection variable has a cursor, which is basically how the database and your application talk to each other. Cursors do things like keep track of where you are in a long list of data, in addition to a bunch of other stuff, but thats the important part.
When you create the collection cursor outside the view method, it will re-use the cursor on each request if it exists. So, if you make one request, and then another really, really fast right after that (like what happened when you applied high load), the cursor might only be half way through talking to the database, and so some of your data goes to the first request, and some to the second. The reason you would get NO data in a request would be if a cursor finished fetching data but hadn't been closed yet, so the next request tried to fetch data from the cursor, and there was none left to fetch in the query.
By moving the collection definition (and by association, the cursor definition) into the view stack, you will ALWAYS get a new cursor when you process a new request. You wont get any cross talking between your cursors and different requests, as each request cycle will have its own.
I am new to Flask. How do I call render_template('prices.html', stock_price) once every minute for a given page that is fed by constantly changing data?
I tried this:
throttle.Throttle(period=60) # 60 second throttle
while True:
stock_price = get_stock_price()
render_template('prices.html', stock_price=stock_price)
throttle.check() # Check if 60 seconds have passed otherwise sleep
The only thing that does work is return render_template(...). Apparently render_template() must be part of a return-statement. Unfortunately, once return is called the game is over.
How do I accomplish this? I'm assuming it is just ignorance on my part.
When you render the template you run the Flask process and return an HTML after the template file rendered, and as your said, you use return and the game is over.
Even if you will find a really hard-code way to do it, it's not a good way to do it. You will waste too much server calls and a user waiting time.
The easy way to do it
Writing a really simple Python function that return only the stock number(i guess you already have one - get_stock_price() )
But i would decorate it with a route(Lets say "/getprice"). Every one who will get to this page will get a black page but the stock price text.
Now for the real magic - use JQuery-AJAX on the HTML page to call this function:
$.ajax({
type: "POST",
url: "/getprice",
})
.done(function( price ) {
$("#price-box").val(price)
});
Hey but wait, what about my "once per minute"??
of curse we can insert this ajax call into setInerval() function like this:
setInterval(function() {
$.ajax({
type: "POST",
url: "/getprice",
})
.done(function( price ) {
$("#price-box").val(price)
});
}, 1000 * 60);
Tell me if you need any help with that.
The Pro way to do it
Ok, so you managed to send a request for your get_stock_price() function once per minute.
Great.
But what about performance?
I guess your get_stock_price() doing some other request or even some web scraping which, again, could be really hard for the server (think what going on when 10,000 users using this page).
What i would do is:
Store the data from get_stock_price() in your DB every minute(Cron Job would do the job), then when a user asks(AJAX request) for this data, pull it out of the DB.
This way the server will work on the background and the user won't see any different with his data loading speed.