I'm using Django for a large app. When monitoring the DB (Postgres) I sometimes see a few slow queries log written.
The problem is how to track down the code that generated these queries. optimally I want some stack trace for these logs, but wonder if there is some other best-practice, or maybe some other tool.
It's in production so DEBUG is set to False, so Django itself doesn't track the speed of the queries.
P.S - I saw somewhere someone suggesting to add SQL comment to the SQL query itself. Unfortuantely this doesn't seem to be generally available in Django, and also it will require me to update every single code that generates queries.
There are plenty of tools out there to inspect of monitor the SQL queries that get executed inside a Django application.
Django Debug Toolbar
The Django Debug Toolbar is a configurable set of panels that display various debug information about the current request/response and when clicked, display more details about the panel's content.
Django Silk
The Django Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection
Elastic APM
With Elastic APM, monitor, detect, and diagnose complex application performance issues to maintain an expected level of service.
This is more like a production setup and you can visualize the matrics very easily with the help of out-of-the-box features. They do have an APM setup section for Django
Related
I'm new to Django (in server side programing in general) and I don't know how to declare a state and be able to update it from client side.
What I want to do is be able for some users to set a "maintenance" state (boolean) to true or false.
I'm using Django Rest Framework.
Can someone give me a clue or documentation to read?
Thanks
Based on the comments under your question I assume we're discussing a maintenance mode of the whole service.
This would usually be done by the service infront of your application, i.e. nginx, which you need for other things as well.
But since you need the site admins to be able to trigger it you could store that value in a cache service.
One of them would be Redis which works nicely with Django's cache framework. You could make a custom middleware that would check for that maintenance value in Redis and throw the maintenance page.
A middleware would be optimal as it will ensure the whole service (all URLs) are kept hidden if maintenance mode is on.
I am working on creating a dashboard which would access Redmine project data. This dashboard will be a consolidation of project data (delivery date, manager, start date, total issues, open issues, closed issues, bug count etc.) for all projects in the Redmine database.
Any ideas on how to do this? An SQL query to achieve this would be a good start. I have Redmine setup on my local system and can access the Redmine database.
I plan on creating an HTML5 .Net based web application. Once I figure out the queries to be used, I would write an IIS (or REST) service to fetch the desired data from the Redmine production database.
Any help/pointers would be deeply appreciated.
Regards,
Pranjal
You have basically three options:
use the REST API
develop a redmine plug-in
do some custom SQL stuff
REST API
The best you could do is to use the redmine REST API. Using it, you can fetch the data directly from redmine without any SQL queries or manipulating the server. It should even work when the redmine server is updated.
Redmine Plug-in
The next best thing is probably to develop a redmine plug-in. Your plug-in can access redmine classes (Project, User, etc.). For example, you could do something like Projects.all.map {|p| [p.id, p.identifier]} to get the id and identifier of all projects in an array. This approach is way more powerful than using the REST API, but you have to modify the redmine server (install and maintain your plug-in). A plug-in is probably more fragile when redmine changes internal structures, than the REST API.
Custom SQL stuff
I do not recommend developing custom SQL solutions, but it is an option. Such a solution is harder to develop, more fragile, harder to set up, but might offer better performance.
You can see the the redmine database structure listed in db/schema.rb. Having the right schema, you could develop your custom SQL queries and return them to your dashboard.
Think about security
Apart from the usual security stuff (which you should consider too :)), remember that redmine has a rich permission system. Consider which object you make (publicly) available. This is most important for the last two options. When using the plug-in approach, you should have a look at the User#allowed_to method.
With the little information I have about your requirements, I have the feeling that you should go for the REST API.
I am very new to web development. I have build a django application. Now what I want to do is log user behavior such as mouse clicking, time spend on certain section of the webpage. I am very new django so my django application is very simple and I want to start exploring how to build an user behavior tracker to my django application. Ideally I like to save all this session information in database/table. Where should I start and any example would be a great help.
What you want is just web analyics. There are lots of ready made systems that can do this for you, so you won't need to make it yourself in Django.
I would recommend Piwik as it's an open source solution. The drawback is that you have to install it yourself, so something like Google Analytics might be easier to set up, just remember that using this option enables google to track your visitors too.
There are a lots more web analytics software. For a good overview check this Wiki page:
https://en.wikipedia.org/wiki/List_of_web_analytics_software
I have an Django application at my work, only available on the internal network.
Currently we import data using Excel, but this is a terrible error prone process and I want to replace it.
I would like to provide a rich web application in Javascript which exposes some, but not all of the data from the main Django application (lookup values for menus). This would run on a server visible to the outside world.
So what is a good approach for this?
Management are concerned about security of making the main Django app available to the outside world, and I would prefer an intermediate tier as well - I think it would be easier to write a small server side app than to go through the current code and make sure it is secure enough to the outside world (I learned Django buildiong this app, so some of the older code is not done according to best practices, but does work as it needs to). I would also like it to hold the new data until someone has checked it looks OK before importing to the main database. (I am the only developer, so there are time considerations).
So two options I can think of just now.
1: Have a small Django app on an external facing server. This can communicate with the main app to get the values required for lookups, and store the input before it gets imported. The tables will essentially mirror the main app and need updated when the main app tables change.
2: Have something similar, but rather than use a database, use the external facing server to contact the REST interface on the internal server. Something like using Django non-relational to get data from the REST interface of the main app. Put an import table in the main database server to store the dats for approval.
Are either of these good / bad approaches?
Any other suggestions?
Are there any good resources for learning about n-tier apps?
If I understand you correctly you want a small Group of trusted users to be able to access an internal database. There is already an Internal Django App accessing that database.
Management is concerned about making this app or an extension of it available to the general Internet.
I think ther concerns are very valid. If you have only a limited set of users accessing the import functionality, push authentication out of the Django Web Application into the HTTP Server / Balancer / Frontend.
For example set up an apache external webserver forcing all access to your Django App beeing encrypted (HTTPS) and authenticated. Users can be authenticated via HTTP-Auth using static files on the server. Password changes / user additions have to be done by an admin logging into the server.
Only after completing this login the Django App with it's own authentication can be accessed. I would opt vor a smale seperate import App instead of extending the main app. This small app could run with reduced permissions on the main database for an defense in depth aproach.
This setup provides you with a litte additional interfaces / points of failures, while maintaining a small attack surface against random Internet users. You can hire a security consultant th audit your apache config and be assured that you locked out the greater Internet and only have to worry about HTTP-Authenticated users.
I have benn running such setups for 15 years by now. Users are annoyed by the double authentication and password saving in Internet Cafes is an issue whith HTTP-Auth but generally it is verry seamless if once set up.
I have a quite slow admin interface in a django application, the application is supported by apache2 and PostgreSQL.
I suspect the problem to be unoptimized sql request but I cannot understand which one. I believe that a request is sent for every row instead of a request for all row.
Is it possible to log every sql requests actually sent to my database?
Thank for your advice.
Use the log_min_duration option in the configuration file:
http://www.postgresql.org/docs/current/static/runtime-config-logging.html#GUC-LOG-MIN-DURATION-STATEMENT
You might also want to install the auto-explain module which will also dump the execution plan of the slow queries to the log file.
More details here: http://www.postgresql.org/docs/current/static/auto-explain.html