I'm am trying to optimise some caching for a complex django project, which is struggling with a large number of sql queries.
I'm using 'johnny-cache' to simplify the caching of the queries. It invalidates cached entries for a 'model' on 'save' of data to the model, and as such is geared to sites which are mainly read-only.
I have seen a 'memcached' django admin app which i have used in the past, though cannot find it atm. I would really like a way of monitoring django's "locmem" cache backend, does anyone have any suggestions?
Related
Previous Django apps I've worked on have used HTML template rendering, and django-debug-toolbar has proven to be a valuable way to analyze ORM SQL queries and find places to optimize.
The current app I'm working on uses Django for django-rest-framework REST endpoints only, and has no HTML views.
Is there a way I can analyze queries in a similar way? I looked into django-debug-toolbar + django-debug-panel (with a Chrome extension), but it's quite a bit out of date, requiring Django 2.1 or earlier.
I am not sure about django-debug-toolbar for API testing, because request are made with AJAx in swagger UI and other browsable tool API.
But you can look Django-silk with is more usable for your case with same information of Django-Debug-Toolbar: https://github.com/jazzband/django-silk
Hope this can help
You can try dj-tracker. It will keep track of all your queries (and can even give hints to optimize them).
I am using the sites framework in django. I have a context processor that gives me access to the sites framework which is stored as a model in the database. I could also store the same values in settings.
Is it more efficient to store and retrieve info like site name from the model or from a property in settings?
When it comes to the sites framework, the sites are cached by default and as fast as your settings.
It would be a good idea to keep your settings module for just settings, not as some sort of ad hoc caching mechanism or a replacement for the sites framework.
The settings are cached by default, the database is not. So the settings are definitely faster.
Both are cacheable however so in the end it doesn't matter too much depending on how you use it. Also, it will probably have a negligible amount of influence.
I like mongodb and django,and there are some frameworks to select:
mongodbengine
django-mongodb-engine
Mongodbengine has good performance and django ORM like api,but when serialize ,it's not supported now.
Django-mongodb-engine is a django backend,you can use it with django ORM.
So django-mongodb-engine is better? and how about its performance?
There are a number of projects out there for incorporating MongoDB with Django but the best one (I believe) is yet to come. Alex Gaynor's Google Summer of Code project is working on a queryset refactor of the Django ORM to allow for non-relational backends. In his last update he stated that they have a working MongoDB backend:
Since this is about the halfway point
of GSOC I'll give a general overview:
we have a working MongoDB backend,
with many implemented features, and a
set of changes to Django itself (that
don't break anything else of course)
that enable this.
http://groups.google.com/group/django-developers/browse_thread/thread/36ed23d7b32ff0fd?pli=1
So if you want to use MongoDB with the Django ORM you can switch to django-norel and use django-mongodb-engine or you can wait for Alex's work to be merged into the trunk.
We use django-mongodb-engine in production and we haven't face any performance issues. It is well known that using django-mongodb-engine means that you'll be tight to the django ORM, checks, fields and so on. But, you're also able to execute raw queries when you simply don't care about django ORM.
django-mongodb-engine started taking some ideas from opensource projects (mongodbengine is one of those)
P.S: One of the things I like more about django-mongodb is the easy integration with django because it allow you to simply switch the DB when you need it (Unless you're using "non-standard fields")
P.S2: I'll open a issue requesting some benchmarks
Update: The issue link
I created a database for a site i'm doing using Django as the admin backend. However because the server where the site is hosted on, won't be able to support Python, I find myself needing to do the front end in PHP and as such i've decided to use CodeIgniter along with Datamapper to map the models/relationship.
However DataMapper requires the tables to be in a specific format for it to work, and Django maps its tables differently, using the App name as the prefix in the table. I've tried using the prefix & join_prefix vars in datamapper but still doesn't map them correctly.
Has anyone used a combination of this? and if so how have the fixed the issue of db table names being inconsistent? Is there anything out there that i can use to make them work together?
----edit: clarification on backend---
Let me clarify: i'm going to be running the admin from a subdomain pointing to a python ready server. However i can't move the main domain name from the php only webserver because of certain constraints/binding contracts the company got itself in. and don't want to use cloaking/masking because of seo purposes.
i'm using the django admin because i'm using some packages to make a pretty/functional admin, such as grappelli for the admin template, along with its editor for editing news stories, etc. also using photologue to manage photos/galleries. etc.
If your problem is simply making Django use the same tables as your other software, use the db_column and db_table parameters in the models.
http://www.djangoproject.com/documentation/models/custom_columns/
Two apparent solutions:
Instead of hacking one or both to work well with each other, emulate the Django admin in PHP/CodeIgniter code. **
Get a server that supports Django. Make the frontend in Django.
Time-wise, either one of those solutions will be less involving than trying to make two different frameworks using different programming languages mesh well together. I can't imagine the future maintenance required to ensure everlasting compatibility and interoperability.
Also, I assume by saying:
I created a database for a site i'm doing using Django as the admin backend
You really mean that you modeled your apps using Django, and that you also intend on administrating the database that has resulted from this modeling in the Django admin. (In which case you already have your Models layer complete and should just try building the rest of the site in Django)
If that's the case then in your models you are going to need to define the exact column names (db_column) that DataMapper will expect, as well as manually define the table names (db_table), including M2M tables.
You may also have to define all of your primary keys manually, if DM expects something named differently.
Also:
If the server can't support Python, where are you going to be running your backend? Different server? Locally? This plan just isn't making a lot of sense.
** I would not suggest trying this. I had been attempting to make a CI backend that actually shared much of the same ideas as Django's admin, before I knew about Django's admin. And of course once discovering Django, I dropped the CI work immediately and continued on with what I have found to be a much more amazing framework that is much faster to develop on.
So as I understand you plan on using Django just because of django-admin, and you are trying to use CI for the actual site because the server runs PHP, right?
So why don't you use framework that generates something like Django's admin but that you can run on your server?
The Symfony Framework has a really nice admin generator, in the spirit of Django's and you might be able to run it on your server. This would save you from the maintainance nightmare that might come later as #jonwd7 answered
What would be the best way to port an existing Drupal site to a Django application?
I have around 500 pages (mostly books module) and around 50 blog posts. I'm not using any 3rd party modules.
I would like to keep the current URLS (for SEO purposes) and migrate database to Django. I will create a simple blog application, so migrating blog posts should be ok. What would be the best way to serve 500+ pages with Django? I would like to use Admin to edit/add new pages.
All Django development is similar, and yours will fit the pattern.
Define the Django model for your books and blog posts.
Unit test that model using Django's built-in testing capabilities.
Write some small utilities to load your legacy data into Django. At this point, you'll realize that your Django model isn't perfect. Good. Fix it. Fix the tests. Redo the loads.
Configure the default admin interface to your model. At this point, you'll spend time tweaking the admin interface. You'll realize your data model is wrong. Which is a good thing. Fix your model. Fix your tests. Fix your loads.
Now that your data is correct, you can create templates from your legacy pages.
Create URL mappings and view functions to populate the templates from the data model.
Take the time to get the data model right. It really matters, because everything else is very simple if your data model is solid.
It may be possible to write Django models which work with the legacy database (I've done this in the past; see docs on manage.py inspectdb).
However, I'd follow advice above and design a clean database using Django conventions, and then migrate the data over. I usually write migration scripts which write to the new database through Django and read the old one using the raw Python DB APIs (while it is possible to tie Django to multiple databases simultaneously, too).
I also suggest taking a look at the available blogging apps for Django. If the one included in Pinax suits your need, go ahead and use Pinax as a starting point.
S.Lott answer is still valid after years, I try to complete the analysis with the tools and format to do the job.
There are many Drupal export tools out of there by now but with the very same request I go for Views Datasource choosing JSON as format. This module is very solid and available for the last version of Drupal. The JSON format is very fast in both parsing and encoding and it's easy to read and very Python-friendly (import json).
Using Views Datasource you can create a node view sorted by node id (nid), show a limited number of elements per page, configure a view path, add to it a filter identifier and pass to it the nid to read all elements until you get an empty JSON response.
When importing in Django you have a wide set of tools as well, starting from loaddata to load fixtures. Views Datasource exported JSON but it's not formatted as Django expects fixtures: you can write a custom admin command to do the import, where you can have the full control of the import flow.
You can start your command passing a nid=0 as argument and then let the procedure read, import and then fetch data from the next page passing simply the last nid read in the previous HTTP request. You can even restrict access to the path on view but you need additional configuration on the import side.
Regarding performance, just for example I parsed and imported 15.000+ nodes in less than 10 minutes via a Django 1.8 custom admin command on an 8 core / 8 GB Linux virtual machine and PostgreSQL as DBMS, logging success and error information into a custom model for each node.
These are the basics for import/export between these two platform, for detailed information I described all the major steps for export from Drupal and then import to Django in this guide.