I have a quite slow admin interface in a django application, the application is supported by apache2 and PostgreSQL.
I suspect the problem to be unoptimized sql request but I cannot understand which one. I believe that a request is sent for every row instead of a request for all row.
Is it possible to log every sql requests actually sent to my database?
Thank for your advice.
Use the log_min_duration option in the configuration file:
http://www.postgresql.org/docs/current/static/runtime-config-logging.html#GUC-LOG-MIN-DURATION-STATEMENT
You might also want to install the auto-explain module which will also dump the execution plan of the slow queries to the log file.
More details here: http://www.postgresql.org/docs/current/static/auto-explain.html
Related
I'm using Django for a large app. When monitoring the DB (Postgres) I sometimes see a few slow queries log written.
The problem is how to track down the code that generated these queries. optimally I want some stack trace for these logs, but wonder if there is some other best-practice, or maybe some other tool.
It's in production so DEBUG is set to False, so Django itself doesn't track the speed of the queries.
P.S - I saw somewhere someone suggesting to add SQL comment to the SQL query itself. Unfortuantely this doesn't seem to be generally available in Django, and also it will require me to update every single code that generates queries.
There are plenty of tools out there to inspect of monitor the SQL queries that get executed inside a Django application.
Django Debug Toolbar
The Django Debug Toolbar is a configurable set of panels that display various debug information about the current request/response and when clicked, display more details about the panel's content.
Django Silk
The Django Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection
Elastic APM
With Elastic APM, monitor, detect, and diagnose complex application performance issues to maintain an expected level of service.
This is more like a production setup and you can visualize the matrics very easily with the help of out-of-the-box features. They do have an APM setup section for Django
So, I am writing a django-mongo app, using mongo for storing files as well, and using the python files mongoforms and mongoengine. Things were going well for a while, but just recently I have started to get 504 errors. I am using nginx -> apache (mod_wsgi) for non-static files.
Other (non-mongo) django apps on the same server are working, as are static html pages served up by ngnix. Also, if I turn off mongodb entirely, I get the normal django debug=True error page, so when mongo is just turned off it seems that nginx, apache, and django are all able to function.
However, with mongo in the picture I am now unable to get a response. I am able to use the command line interface to mongo, and although I'm only using the 32b version, it says that I'm only using 0.0625GB for the database in question.
So, I realize this isn't enough information to figure out what my problem is, but that's more or less what I'm looking for: where do I look next? I can connect to mongo, but not via django, although I used to be able to. I've tried commenting out virtually the entire django view, just leaving the index call which is not even asking for anything from mongo, so what I'm looking for is where do I look next? Are there any debugging techniques for mongo where I can figure out why it's taking so long to respond to django? Any other ways to debug this?
I don't expect anyone to miraculously solve my problem here but telling me what info to look for next would help as I don't have much experience debugging the django-mongo connection. It worked at one time but it has stopped working and I'm not sure where to look.
I've a huge django project and have to use Instagram API and its subscriptions model to work. For the subscriptions, my server has to be very responsive and be ready to work asynchronously to set up a hook so as to receive notifications once the user posts. Or that's what the documentation suggests. Now will it be a good thing to use Tornado there? Just for that small part or can I do it using Django in an effective way? if so, how?
You can use the WSGI container on top of Tornado to host any WSGI application, including Django, however, when you do that the WSGI application is still running as a blocking application and will not magically be running as an asynchronous application. So, when Django is handling a request there is no ability to handle another request at the same time within Django. The solution at that point is not much different to running a single threaded WSGI server and you would need to have multiple Tornado instances to handle concurrent requests.
So all really depends on what you mean by asynchronous. You certainly can't make use of Tornado's direct asynchronous programming API in Django. Thus there isn't really any great benefit from using Tornado with Django via the WSGI interface.
As I understand you are talking about this paragraph in Instagram docs
You should build your system to accept multiple update objects per payload - though often there will be only one included. Also, you should acknowledge the POST within a 2 second timeout--if you need to do more processing of the received information, you can do so in an asynchronous task.
That's another type of "asynchronous" that Tornado provides.
I think Django + Celery will suite better for this.
Your application will work in this way:
You receive JSON-data from Instagram
Create a celery-task, e.g. instagram_process.delay(request.raw_post_data) or instagram_process.delay(request.body) according to your Django version
Response to Instagram with 200 status code
In instagram_process task you do all your procession - parse JSON, store it do database and anything else you need.
If you want to check X-Hub-Signature you can either do it between steps 1 and 2, or pass this header to the task and verify the signature at step 4.
You can use tornado.wsgi to integrate Tornado with other WSGI compliant frameworks. Check out this demo project for details:
https://github.com/bdarnell/django-tornado-demo
I have built a website in django framework. It has a lot of features such as blog, discussion forum, basically there are lots of ways users can interact. I have built a basic notification framework where a user gets notified when somebody comments on their blog, or answers their question in the forum.
Since the notifications are stored in db, new notifications are displayed only when a page refresh is done. I would like to make it real time using some push server using something like long polling technique.
I have come across NowJS which seems to be pretty handy for this, but in all the examples that are given I could not see any example where there was any interaction with the database. In all the cases there was some information sent by one client and it was displayed to one or more clients.
What I actually want to do is to call a function using NowJS, and make it go to sleep until a new notification is added in the database. When a new notifications comes in the server responds back with the notification and a new request is done immediately.
I can figure out all other parts except how to access the database from Node server that is used by NowJS. Any help or guidance is appreciated.
Either:
Have your node.js server make an http call to the Django server via something like a REST api to get info back
Google for a database connector for node.js - I found enough evidence for a MySQL one, and rumours of a PostgreSQL one. Note this won't get you access to the Django DB API, so you'll have to work out all your related queries and craft your SQL by hand (make sure Bobby Tables doesn't bother you: http://www.xkcd.com/327)
Re-implement the NowJS protocol so that you can write a django server for it, keeping the same JS client code on the clients... but then you may as wel.....
...use django-socketio http://blog.jupo.org/2011/08/13/real-time-web-apps-with-django-and-websockets/
I am having a django app which was hosted on apache2. The webapp basically makes a request to a server using thrift and renders the output on the webpage. I notice that webapp is really slow. I am not sure if it is machine problem or the API problem. I verified the API's and they are returning responses in few milliseconds. I am not sure, if django is the problem. Is there a way to profile the webapp. I am using python 2.5.2.
Please help.
Thank you
the simplest thing would be to enable logging, if you are using the latest django 1.3, it is nicely integrated with the python logging module, see:
http://docs.djangoproject.com/en/1.3/topics/logging/
here you can define a Formatter which saves the time each log message is written, see example in django docs:
http://docs.djangoproject.com/en/1.3/topics/logging/#an-example