I have a CMS running on Django 1.4 and the database is Postgresql 9.1. I have a lot of content in the CMS and the issue I am facing right now is that the Django Admin search takes forever to retrieve the results. I would like to know if there are options to optimize this behavior of Django Admin search. I know that Django uses LIKE query on Postgresql for doing lookups. I know that Postgresql 9.1 has the GIN and GIST Index which could help to speed up this behavior of Django. I can also modify this search behavior to make it fast and compromise a little on the quality of search results. I would like to know the most optimum approach to optimize this search behavior of Django?
You might want to use the Django Debug toolbar to check which SQL queries are actually slow.
We found that Django admin's implicit use of UPPER resulted in Postgres ignoring all the existing indices. If that's the problem you could create an index on the uppercase representation of your data.
If you do not want to alter Django, profile the search query and add appropriate GIN and GIST indexes. Otherwise, you may want to look at integrating something like Haystack to power the search a bit faster and without tying up your database.
Helpful Links
http://www.rossp.org/blog/2009/jan/28/django-postgresql-fulltext/
Related
Working on a Django project that requires a NoSQL store and I believe I've settled on Mongo. I've seen a lot of topics that talk about Mongo and Django, but none that mentioned Django-nonrel, and I can't see why it might have been disqualified, but I have no experience with any of them.
Ideally, I'd like to preserve the SQL store for the simple things, user auth, groups, etc., and use Mongo for the bigger data.
I'd also like to have my Mongo-stored objects as classes, in a Django-ORM style, so that I can have a similar 'feel', but that isn't critical.
Lastly, do any of the above allow me to use Django's multi-database support for anything, or are all my mongo queries effectively 'out of band' from the Django ORM?
If there are any other factors I'm missing that might be important, feel free to clue me in. I know little in this arena.
Django-nonrel is the way to go for Django on MongoDB. There's django-mongodb.org, but that's just built on top of Django-nonrel. There's quite a bit of mongodb activity going on the django-nonrel mailing list.
Storing your mongo classes as Django ORM objects works fine, that's the whole point.
I haven't tried using the multi-database support along with SQL. I haven't seen many people use it like that, and I suspect it most likely does not work. There's some work on moving django-nonrel to be officially part of Django 1.4, I suspect it'll work after that's complete.
Using django-nonrel for auth works ok. The main problem is many-to-many relations. The auth module uses that for per user object permissions - that doesn't work. If you don't need that, you could probably get away without using the SQL at all.
Adding onto dragonx answer. The issue with django-nonrel is that the auth module doesn't work.
You can perform many-to-mamy joins using $lookup operator. djongo does automatically for you. It translates SQL syntax into mongodb aggregation queries and populates the object model, like other SQL drivers.
The auth module works fine on djongo
I am using SOLR with NUTCH
I have successfully configure both and Solr is giving me the desired results
Now i want to integrate this in my Django project
Can anyone suggest me how to do it ...
it depends on what you need:
IF YOUR SOLR INSTANCE IS BOUND TO THE DJANGO'S MODEL
you probably are looking for django-haystack: it's pretty easy to setup and to use inside your django views, but it's not suited for huge amount of data
IF YOU'RE USING SOLR WITHOUT ANY RELATION TO DJANGO MODELS
this means that your results are retrieved as pure array, and you just need a python interface to solr. you have some choices
pysolr
solrpy
sunburnt (i'm using this)
you have to try them out and find out which one satisfies you more (check their websites too, to see if the project is mantained. also, stumble into their source code, to check if there's all you need). being a python interface implies that you have to do a little bit more of work inside your views
i hope this can help you
I'm in the process of choosing a Framework for a new project. I have basically the database schema developed(I have this schema running in other PHP webapps already).
In this project I will need to basically search the database schema with Solr. The database schema is a little bit complex to define models in Django, so I think the only option I have is to execute SQL directly... my doubt is about Haystack/Solr... It is possible to query Haystack/Solr when I have no Django Models defined?
PS: I'm new to Django, I have never userd Haystack.
Haystack is pretty tightly coupled to the Django ORM. If you're not using Django models, I don't think Haystack is suitable. I've only used Haystack briefly, so I might wrong.
From the Haystack docs:
When should I not be using Haystack?
Non-Model-based data. If you just want to index random data (flat files, alternate sources, etc.), Haystack isn’t a good solution. Haystack is very Model-based and doesn’t work well outside of that use case.
I never used haystack, but you can always perform raw sql queries.
Have a look on documentation:
https://docs.djangoproject.com/en/dev/topics/db/sql/#executing-custom-sql-directly
For a project I implemented Haystack but now I was wondering what the pros and cons are of using Haystack over the ORM. For my project I need to find books by their title and isbn. I guess Haystack is more useful if you have to do full text searches or am I missing something?
Haystack is certainly better for full text search, and also provides a bunch of other benefits that you won't get using the ORM. Here are some from the Haystack site:
Spelling suggestions
Faceting
More like this
Highlighting
Search term boosting
Besides that, you should use Haystack over the ORM if you expect a high volume of searches, since you'll offload all that work from the DB (which should be focused on writing and retrieving your data).
http://haystacksearch.org/
The main advantage to use haystack over the ORM is performance. LIKE requests are just going to kill you database if you have 100 concurrent users searching. If you have MYSQL of course you can still use its full text capability, but it will still be slower than Xapian or Solr.
Plus you will have additional features such as fuzzy searching that user loves a lot.
It does comes with the extra work of setting up solr and maintaining it, so:
if your website is going to be small forever, like for small company intranet, don't bother and user ORM.
if you plan for the big bad Web, go Haystack.
And not forget to mention that doing a basic haystack setup isn't really much more work than doing a search view using Django's ORM.
Haystack is a module that enables easier search integrations for your django application
you can create something similiar to a ModelForm, called ModelIndex, in this new class you will be able to declare the searchable fields of your models and other settings
It has little to do with Django ORM (the orm is used to comunicate with your databases)
of course you can have the orm query your db models with the search pattern specified, but Haystack is a better choice for setting up a search engine easily on your website, so... if you have to enable complex searches on your site, just go with Haystack
I like mongodb and django,and there are some frameworks to select:
mongodbengine
django-mongodb-engine
Mongodbengine has good performance and django ORM like api,but when serialize ,it's not supported now.
Django-mongodb-engine is a django backend,you can use it with django ORM.
So django-mongodb-engine is better? and how about its performance?
There are a number of projects out there for incorporating MongoDB with Django but the best one (I believe) is yet to come. Alex Gaynor's Google Summer of Code project is working on a queryset refactor of the Django ORM to allow for non-relational backends. In his last update he stated that they have a working MongoDB backend:
Since this is about the halfway point
of GSOC I'll give a general overview:
we have a working MongoDB backend,
with many implemented features, and a
set of changes to Django itself (that
don't break anything else of course)
that enable this.
http://groups.google.com/group/django-developers/browse_thread/thread/36ed23d7b32ff0fd?pli=1
So if you want to use MongoDB with the Django ORM you can switch to django-norel and use django-mongodb-engine or you can wait for Alex's work to be merged into the trunk.
We use django-mongodb-engine in production and we haven't face any performance issues. It is well known that using django-mongodb-engine means that you'll be tight to the django ORM, checks, fields and so on. But, you're also able to execute raw queries when you simply don't care about django ORM.
django-mongodb-engine started taking some ideas from opensource projects (mongodbengine is one of those)
P.S: One of the things I like more about django-mongodb is the easy integration with django because it allow you to simply switch the DB when you need it (Unless you're using "non-standard fields")
P.S2: I'll open a issue requesting some benchmarks
Update: The issue link