For performance monitoring I want to send the count of total queries and the elapsed query time for each request to statsd. In dev/staging I can use connection.queries but I obviously don't want to enable debug on production.
Is there another way to do this without writing a custom cursor wrapper? Anyone have experience doing this?
Use django debug toolbar specifically designed for debugging purposes. It display various debug information about the current request/response and when clicked, display more details about the panel's content.
Related
I see time of query QUERY = 'SELECT COUNT_BIG(*) AS [__count] FROM ... in Django debug toolbar. Is it pure perfomance of database or dirty time that includes handling of query by Django and third-party libraries?
As seen on the docs:
SQL
classdebug_toolbar.panels.sql.SQLPanel
SQL queries including time to execute and links to EXPLAIN each query.
This means it doesn't include django processing time.
https://django-debug-toolbar.readthedocs.io/en/latest/panels.html#sql
I have an Apex application that is quite large. The need has come up to store detailed usage logs of this application. The information on APEX_WORKSPACE_ACTIVITY_LOG is not enough, because I need to know what queries each user runs on each page.
My first thought was to get the actual Oracle query logs (V$SQL and such), but they provide no information on the user (as far as the database is concerned, all queries are made by APEX_PUBLIC_USER). I have some information about the user on V$ACTIVE_SESSION_HISTORY, but that's incomplete because it stores samples of active sessions and their SQL queries at 1-second intervals, so I miss too many queries.
So now I'm off to implementing application level logging. The "right" way to fo this would be to go through all the pages in my application and create a logging process to store the relevant information for each one (username and some page items). But I wonder if there might be something simpler that does the trick.
If I understand correcly, "application processes" are run by every page in the application. So if I can get an application process to iterate over the list of page items, I can store them all in the database and be done with it. Something like
for item in page_items {
log(username, item_name, item, date)
}
Can this be done? Or maybe the information I need is on the database already and I don't see it?
You can query the metadata tables to get all items for a specific page and then use that to get their value.
select item_name, v(item_name) item_value
from apex_application_page_items
where application_id = :APP_ID
and page_id = :APP_PAGE_ID;
That will capture all items on the page. Don't forget that if you use items on Page 0 (Global Page) you may want to query that page too.
Additionally, you may want to capture application level items too.
select item_name, v(item_name) item_value
from apex_application_items
where application_id = :APP_ID;
I'm currently in the process of optimizing my Django app, which is acting as an API for my front-end with the Django Rest Framework. While running my server in debug mode, I've noticed that every time a queryset gets executed, there's a query run right before it that always looks like this:
SELECT COUNT('*') AS "__count" FROM "table_name WHERE ..."
The ... part always mirrors the query that returns the objects that I want. I'm unsure if this is only run in debug, something that the QuerySet object does innately, or an error with my code. Would appreciate some insight as to why this is happening and if it's something I need to worry about
This occurs in Django Rest Framework when you are using paging on a list view:
One query to fetch the data for your current page.
A second query to calculate the total number of records for the same queryset.
I will have a sidebar that appears on almost every page of my web app. The sidebar will have some drop-downs, which will consist of a total of say, 1000 different options, which are pulled from the db. Rather than doing the query to get these choices on every page load, I think it makes more sense to just do the query once, say in my config.py and store them in a variable that my views have access to. Is this OK? Is there a better way to accomplish this?
You could do that, but then you'd need to restart your Flask server every time you wanted to update the sidebar.
I'd use some other form of caching here. You could use Flask-Cache and memoize your query results. You can pick a nice long cache timeout, and then clear the cached result whenever you update the sidebar.
I'm thinking of creating a log system for my django web application. The web application is quite comprehensive in its use (covers all aspects of a business's processes) so I'd like to track every event that happens. Specifically, I'd like to log every view that runs and not just the "main" ones and, potentially, log what is happening within the view as its executed.
While I'm in the "idea" stage of the logging system, I've quickly hit a few questions that leave me unsure how to proceed. Here are the main questions I have:
I'm thinking of logging all of the events in the same MySQL database that the main web app holds its data. The concern I have is bloating the MySQL database into a massive DB. Also, if the DB crashes or is destroyed somehow (yes I have backups) I'll loose my log too which blows away any ability to track down the problem. Do I use a seperate DB or just go with text files?
How granular do I go? Initially I was thinking of simply logging things like, "Date - In view myView". However, as I'm thinking about it, it would be nice to log all the stuff that happens within the view. Doing this could make the log massive! and would also make my code ugly with so many log entry lines mixed into the code. This kind of detail:
Date - entered view myView
Date - in view myView, retrieved object myObject from the DB
Date - in view myView, setting myObject field myField to myNewValue
Date - leaving myView
Those are my main thoughts at this point. Any advice on this front?
Thanks
I think the best and right way is to create your own custom middleware where you can log actually everything you need.
Here are some links on the subject:
middleware snippets
http://djangosnippets.org/snippets/2624/
http://djangosnippets.org/snippets/290/
http://djangosnippets.org/snippets/264/
django-logging-middleware (pretty old by may give you an idea)
django-request
django.db.backends logging
Is there a Django middleware/plugin that logs all my requests in a organized fashion?
Django verbose request logging
log all sql queries
django orm, how to view (or log) the executed query?
Also, consider using sentry error logging and aggregation platform instead of writing logs into the database. FYI, see using a database for logging.
If you want to log any action run in every view, you can for example, replace entered view A and exited view A by a line in these words: view A - 147ms.
As alecxe stated, you can log requests/SQL, there are plenty of ways to do it with middleware. About database (object) actions, you can tie individual saves, updates and deletes with signals.
For bulk updates and deletes, you could (it's not a clean way but it would work) monkey-patch manager and queryset methods to add logging.
This way you can log actions rather than SQL.
I would see lines like this:
[2013/09/11 15:11:12.0153] view app.module.view 200 148ms
[2013/09/11 15:11:12.0189] orm save:auth.User,id=1 3ms
This is a quick and dirty proposal, but, maybe it's worth it.