I'm currently in the process of optimizing my Django app, which is acting as an API for my front-end with the Django Rest Framework. While running my server in debug mode, I've noticed that every time a queryset gets executed, there's a query run right before it that always looks like this:
SELECT COUNT('*') AS "__count" FROM "table_name WHERE ..."
The ... part always mirrors the query that returns the objects that I want. I'm unsure if this is only run in debug, something that the QuerySet object does innately, or an error with my code. Would appreciate some insight as to why this is happening and if it's something I need to worry about
This occurs in Django Rest Framework when you are using paging on a list view:
One query to fetch the data for your current page.
A second query to calculate the total number of records for the same queryset.
Related
I see time of query QUERY = 'SELECT COUNT_BIG(*) AS [__count] FROM ... in Django debug toolbar. Is it pure perfomance of database or dirty time that includes handling of query by Django and third-party libraries?
As seen on the docs:
SQL
classdebug_toolbar.panels.sql.SQLPanel
SQL queries including time to execute and links to EXPLAIN each query.
This means it doesn't include django processing time.
https://django-debug-toolbar.readthedocs.io/en/latest/panels.html#sql
In my Django code, some postgres queries are created by the django query set. One of the queries, which can be created by two different view functions in the home page is being done during runtime. I don't know which one is doing the query. For debuggin purposes Is there anyway to trace back a postgres query to which line of code generated it during runtime?
I'm thinking of creating a log system for my django web application. The web application is quite comprehensive in its use (covers all aspects of a business's processes) so I'd like to track every event that happens. Specifically, I'd like to log every view that runs and not just the "main" ones and, potentially, log what is happening within the view as its executed.
While I'm in the "idea" stage of the logging system, I've quickly hit a few questions that leave me unsure how to proceed. Here are the main questions I have:
I'm thinking of logging all of the events in the same MySQL database that the main web app holds its data. The concern I have is bloating the MySQL database into a massive DB. Also, if the DB crashes or is destroyed somehow (yes I have backups) I'll loose my log too which blows away any ability to track down the problem. Do I use a seperate DB or just go with text files?
How granular do I go? Initially I was thinking of simply logging things like, "Date - In view myView". However, as I'm thinking about it, it would be nice to log all the stuff that happens within the view. Doing this could make the log massive! and would also make my code ugly with so many log entry lines mixed into the code. This kind of detail:
Date - entered view myView
Date - in view myView, retrieved object myObject from the DB
Date - in view myView, setting myObject field myField to myNewValue
Date - leaving myView
Those are my main thoughts at this point. Any advice on this front?
Thanks
I think the best and right way is to create your own custom middleware where you can log actually everything you need.
Here are some links on the subject:
middleware snippets
http://djangosnippets.org/snippets/2624/
http://djangosnippets.org/snippets/290/
http://djangosnippets.org/snippets/264/
django-logging-middleware (pretty old by may give you an idea)
django-request
django.db.backends logging
Is there a Django middleware/plugin that logs all my requests in a organized fashion?
Django verbose request logging
log all sql queries
django orm, how to view (or log) the executed query?
Also, consider using sentry error logging and aggregation platform instead of writing logs into the database. FYI, see using a database for logging.
If you want to log any action run in every view, you can for example, replace entered view A and exited view A by a line in these words: view A - 147ms.
As alecxe stated, you can log requests/SQL, there are plenty of ways to do it with middleware. About database (object) actions, you can tie individual saves, updates and deletes with signals.
For bulk updates and deletes, you could (it's not a clean way but it would work) monkey-patch manager and queryset methods to add logging.
This way you can log actions rather than SQL.
I would see lines like this:
[2013/09/11 15:11:12.0153] view app.module.view 200 148ms
[2013/09/11 15:11:12.0189] orm save:auth.User,id=1 3ms
This is a quick and dirty proposal, but, maybe it's worth it.
Hi I have some problems that has been bothering me for a week. I am running Selenium testing scripts on my dev machine, and in my test I would call simple script to delete accounts by their sub domain names:
for a in Account.objects.filter(domain = sub_domain):
a.delete()
The problem is that the query to find all such accounts are not returning correct results after the first time it is run (I use this query to clean up the database before each test). When I set a break point at this point, I can see the query return 0 records, even though in the database it has one record. I also set up mysql query log to see the actual query Django sent to mysql, and the query looks good, and will return correct result if I copy and paste to mysql command shell.
What am I missing? Why Django model query does not give me the correct result? MySQL is using InnoDB engine in case that makes any difference.
Transactions. Do a COMMIT in the shell.
This is a recurring problem, so I'm doing a shameless plug with a question in which I described details of the problem:
How do I deal with this race condition in django?
I'm using the development server (runserver) in Django and it's started to annoy me that Django is validating the models every time I save a file and the server restarts. I have ~8000 entries in my SQLite3 database and it takes up to five seconds to validate the models. I'm not familiar with how Django validates the models, but I'm guessing it's proportional to the size of the database somehow.
So is there any way to tell Django not to validate the models? The ideal thing would be being able to tell Django to validate the models only on the first start and not on any automatic restarts due to changes in the Python files.
Django doesn't do any model-level validation at all, and it certainly doesn't scan your database on startup.
The only validation it does on startup is to check the syntax of your models code, and that's not at all proportional to your database size.
I have over 10 million rows in my database, and runserver takes less than a second.
Try hitting Control + C while it is in the 5 seconds, and see where the code is when the KeyboardException is thrown.