I see time of query QUERY = 'SELECT COUNT_BIG(*) AS [__count] FROM ... in Django debug toolbar. Is it pure perfomance of database or dirty time that includes handling of query by Django and third-party libraries?
As seen on the docs:
SQL
classdebug_toolbar.panels.sql.SQLPanel
SQL queries including time to execute and links to EXPLAIN each query.
This means it doesn't include django processing time.
https://django-debug-toolbar.readthedocs.io/en/latest/panels.html#sql
Related
In my Django code, some postgres queries are created by the django query set. One of the queries, which can be created by two different view functions in the home page is being done during runtime. I don't know which one is doing the query. For debuggin purposes Is there anyway to trace back a postgres query to which line of code generated it during runtime?
I'm currently in the process of optimizing my Django app, which is acting as an API for my front-end with the Django Rest Framework. While running my server in debug mode, I've noticed that every time a queryset gets executed, there's a query run right before it that always looks like this:
SELECT COUNT('*') AS "__count" FROM "table_name WHERE ..."
The ... part always mirrors the query that returns the objects that I want. I'm unsure if this is only run in debug, something that the QuerySet object does innately, or an error with my code. Would appreciate some insight as to why this is happening and if it's something I need to worry about
This occurs in Django Rest Framework when you are using paging on a list view:
One query to fetch the data for your current page.
A second query to calculate the total number of records for the same queryset.
I'm thinking of creating a log system for my django web application. The web application is quite comprehensive in its use (covers all aspects of a business's processes) so I'd like to track every event that happens. Specifically, I'd like to log every view that runs and not just the "main" ones and, potentially, log what is happening within the view as its executed.
While I'm in the "idea" stage of the logging system, I've quickly hit a few questions that leave me unsure how to proceed. Here are the main questions I have:
I'm thinking of logging all of the events in the same MySQL database that the main web app holds its data. The concern I have is bloating the MySQL database into a massive DB. Also, if the DB crashes or is destroyed somehow (yes I have backups) I'll loose my log too which blows away any ability to track down the problem. Do I use a seperate DB or just go with text files?
How granular do I go? Initially I was thinking of simply logging things like, "Date - In view myView". However, as I'm thinking about it, it would be nice to log all the stuff that happens within the view. Doing this could make the log massive! and would also make my code ugly with so many log entry lines mixed into the code. This kind of detail:
Date - entered view myView
Date - in view myView, retrieved object myObject from the DB
Date - in view myView, setting myObject field myField to myNewValue
Date - leaving myView
Those are my main thoughts at this point. Any advice on this front?
Thanks
I think the best and right way is to create your own custom middleware where you can log actually everything you need.
Here are some links on the subject:
middleware snippets
http://djangosnippets.org/snippets/2624/
http://djangosnippets.org/snippets/290/
http://djangosnippets.org/snippets/264/
django-logging-middleware (pretty old by may give you an idea)
django-request
django.db.backends logging
Is there a Django middleware/plugin that logs all my requests in a organized fashion?
Django verbose request logging
log all sql queries
django orm, how to view (or log) the executed query?
Also, consider using sentry error logging and aggregation platform instead of writing logs into the database. FYI, see using a database for logging.
If you want to log any action run in every view, you can for example, replace entered view A and exited view A by a line in these words: view A - 147ms.
As alecxe stated, you can log requests/SQL, there are plenty of ways to do it with middleware. About database (object) actions, you can tie individual saves, updates and deletes with signals.
For bulk updates and deletes, you could (it's not a clean way but it would work) monkey-patch manager and queryset methods to add logging.
This way you can log actions rather than SQL.
I would see lines like this:
[2013/09/11 15:11:12.0153] view app.module.view 200 148ms
[2013/09/11 15:11:12.0189] orm save:auth.User,id=1 3ms
This is a quick and dirty proposal, but, maybe it's worth it.
For performance monitoring I want to send the count of total queries and the elapsed query time for each request to statsd. In dev/staging I can use connection.queries but I obviously don't want to enable debug on production.
Is there another way to do this without writing a custom cursor wrapper? Anyone have experience doing this?
Use django debug toolbar specifically designed for debugging purposes. It display various debug information about the current request/response and when clicked, display more details about the panel's content.
Hi I have some problems that has been bothering me for a week. I am running Selenium testing scripts on my dev machine, and in my test I would call simple script to delete accounts by their sub domain names:
for a in Account.objects.filter(domain = sub_domain):
a.delete()
The problem is that the query to find all such accounts are not returning correct results after the first time it is run (I use this query to clean up the database before each test). When I set a break point at this point, I can see the query return 0 records, even though in the database it has one record. I also set up mysql query log to see the actual query Django sent to mysql, and the query looks good, and will return correct result if I copy and paste to mysql command shell.
What am I missing? Why Django model query does not give me the correct result? MySQL is using InnoDB engine in case that makes any difference.
Transactions. Do a COMMIT in the shell.
This is a recurring problem, so I'm doing a shameless plug with a question in which I described details of the problem:
How do I deal with this race condition in django?