Using Django & Haystack with ElasticSearch.
After installing haystack and ES, and Rebuilding Index
./manage.py rebuild_index
WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
Your choices after this are to restore from backups or rebuild via the rebuild_index command.
Are you sure you wish to continue? [y/N] y
Removing all documents from your index because you said so.
All documents removed.
Indexing 1039 <django.utils.functional.__proxy__ object at 0x10ca3ded0>.
AttributeError: 'module' object has no attribute 'ElasticSearchError'
Updating index has the same problem
/manage.py update_index
Indexing 1039 <django.utils.functional.__proxy__ object at 0x10ea49d90>.
AttributeError: 'module' object has no attribute 'ElasticSearchError'
Clear index works fine though ( probably because there is no index )
./manage.py clear_index
WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
Are you sure you wish to continue? [y/N] y
Removing all documents from your index because you said so.
All documents removed.
Versions
django-haystack==2.0.0-beta
pyelasticsearch==0.5
elasticsearch==0.20.6
localhost:9200 says :
{
"ok" : true,
"status" : 200,
"name" : "Jigsaw",
"version" : {
"number" : "0.20.6",
"snapshot_build" : false
},
"tagline" : "You Know, for Search"
}
Haystack settings :
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
},
}
search_indexes.py :
import datetime
import haystack
from haystack import indexes
from app.models import City
class CityIndex(indexes.SearchIndex, indexes.Indexable):
text = indexes.CharField(document=True, use_template=True)
name = indexes.CharField(model_attr='name')
state = indexes.CharField(model_attr='state')
country = indexes.CharField(model_attr='country')
lat = indexes.FloatField(model_attr='latitude')
lon = indexes.FloatField(model_attr='longitude')
alt = indexes.FloatField(model_attr='altitude')
pop = indexes.IntegerField(model_attr='population')
def get_model(self):
return City
Any help - why I am getting error ?
Solved it !
After debugging the process using pdb
./manage.py rebuild_index
At line 222 - in /haystack/backend/elasticsearch_backend.py
Changed
except (requests.RequestException, pyelasticsearch.ElasticSearchError), e:
To
# except (requests.RequestException, pyelasticsearch.ElasticSearchError), e:
except Exception as inst:
import pdb; pdb.set_trace()
I found out the core error was this
'ElasticSearch' object has no attribute 'from_python'.
To which I found solution here - https://github.com/toastdriven/django-haystack/issues/514#issuecomment-4058230
The version of pyelasticsearch I was using was from http://github.com/rhec/pyelasticsearch,
So I installed pyelasticsearch from a fork - http://github.com/toastdriven/pyelasticsearch using :
pip install --upgrade git+https://github.com/toastdriven/pyelasticsearch.git#3bfe1a90eab6c2dfb0989047212f4bc9fb814803#egg=pyelasticsearch
and That fixed it & Index was build !
Related
How can I custom the django_migrations table name?
Even I edit the
venv.lib.site-packages.django.db.migrations.recorder > MigrationRecorder > meta = db_table = "ABC_django_migrations"
the makemigrations cannot detect the changes.
I am using Django version: 3.0.5
Django migrations don't look for changes in its own migration model
MigrationExecutor just ensures that following table exists in database
def migrate(self, targets, plan=None, state=None, fake=False, fake_initial=False):
self.recorder.ensure_schema()
....
where ensure_schema() just creates the table
def ensure_schema(self):
"""Ensure the table exists and has the correct schema."""
# If the table's there, that's fine - we've never changed its schema
# in the codebase.
if self.has_table():
return
# Make the table
try:
with self.connection.schema_editor() as editor:
editor.create_model(self.Migration)
except DatabaseError as exc:
raise MigrationSchemaMissing("Unable to create the django_migrations table (%s)" % exc)
you can manually make migration to edit this model ( AlterModelTable or custom sql) but I would not advise changing anything regarding migrations
How I solve this problem:
Install the app
pip install django-db-prefix
Include the apps in Settings.py
INSTALLED_APPS = ['django_db_prefix',]
Adding the prefix in Settings.py
DB_PREFIX = "foo_"
I was running this file fine yesterday and now I'm getting an AttributeError when I tried running it today. Here is the code I'm trying to run:
from openpyxl import load_workbook
def read_in():
wb = load_workbook('GenerateModel.xlsx')
ws = wb.get_sheet_by_name('Sheet1')
da_name = []
for i in range(1, ws.max_row+1):
if ws.cell(row=i,column=3).value != None and (
ws.cell(row=i,column=3).value != u'DA Name'):
da_name.append(ws.cell(row=i,column=3).value.encode('ascii'))
start_date = ws.cell(row=4, column=4).value
end_date = ws.cell(row=4, column=5).value
if start_date == None or end_date == None:
raise ValueError('Date cannot be left blank')
if start_date > end_date:
raise ValueError('Start Date must be less than End Date')
And here is the error that I get:
line 28, in read_in
for i in range(1, ws.max_row+1):
AttributeError: 'Worksheet' object has no attribute 'max_row'
I tried running another python script and I'm getting an ImportError that says
from openpyxl.styles import PatternFill
ImportError: cannot import name PatternFill
This makes me think that there is something wrong with the openpyxl module. I installed it using pip and I'm using the Spyder IDE. Thanks for the help.
It says '1.8.5'
Your openpyxl version is seriously outdated - hence the absence of max_row attribute on a WorkSheet class.
You need to figure out what Python executable is used when you run your script from the Spyder IDE and then update openpyxl in that environment:
pip install --upgrade openpyxl
(currently latest openpyxl is 2.4.8)
I am getting issue with " raise BulkIndexError while running this python manage.py rebuild_index ?
Here is my haystack configuration in settings.py file
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
#'SILENTLY_FAIL': False,
},
}
HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'
Here is my search_indexes.py
class ProductIndex(indexes.SearchIndex, indexes.Indexable):
text = indexes.CharField(document=True, use_template=True)
content_auto = indexes.EdgeNgramField(model_attr='title')
def get_model(self):
return Product
def index_queryset(self, using=None):
return self.get_model().objects.all()
here is my views.py
def search_titles():
products = SearchQuerySet().autocomplete(content_auto=request.POST.get('search_text', ''))
return render_to_response('sea.html', {'products':products})
When I try to indexing the my product model I ran this command
python manage.py rebuild_index
, It has not indexed. It raises
File "/home/Documents/swamy/project/env/local/lib/python2.7/site-packages/elasticsearch/helpers/init.py", line 156, in streaming_bulk
raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)
elasticsearch.helpers.BulkIndexError: ('500 document(s) failed to index.'
this error.
My model has 21000 products, Can any one help to fix this issue ?
Thanks In Advance !
Elasticsearch changed the way bulk indices are created.
You can use version 1.4.0 which works seamlessly with django-haystack.
I have a database "tumblelog" (using mongoengine) in which I added some data in the "user" collection with a "User" model:
db.user.find()
...
{ "_id" : ObjectId("4fb0c9494ca88402dd000000"), "_types" : [ "User" ], "first_name" : "John", "last_name" : "Doe", "_cls" : "User", "email" : "jdoe#example.com" }
{ "_id" : ObjectId("4fb0cb9d4ca88402ec000000"), "_types" : [ "User" ], "first_name" : "Joe30", "last_name" : "Foe", "_cls" : "User", "email" : "hu#huu.com" }
When I try User.objects in a django shell, I get the following error:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File ".../mongoengine/queryset.py", line 1127, in __getitem__
return self._document._from_son(self._cursor[key])
File ".../mongoengine/base.py", line 1018, in _from_son
obj = cls(**data)
TypeError: __init__() keywords must be strings
Same thing when I try
for user in User.objects:
print user.first_name
---- Edit ----
I tried this
>>> users = User.objects
>>> users.count()
7
>>> users.first()
...
TypeError: __init__() keywords must be strings
---- Edit 2 ----
I installed my project this way :
> virtualenv Test
> source Test/bin/activate
> pip install Django
> pip install mongoengine
> cd Test/
> django-admin.py startproject db
Then I added the lines
from mongoengine import *
connect('tumblelog')
in settings.py
then I created this simple model
from mongoengine import *
class User(Document):
email = StringField(required=True)
name = StringField(max_length=50)
then I run the server
> python manage.py runserver
And in the shell (python manage.py shell) I can save data if I import my model class but I can't read it, I always have the same TypeError: init() keywords must be strings !
-----Switching to django-mongodb engine----
I didn't find any solution so I will use django-mongodb-engine. I did not find any comparison, but I tried both and it's very similar.
I just regret django-mongodb-engine doesn't handle inheritance principle.
What am I doing wrong ?
Thanks in advance!
we had the exact same issue using mongoengine 0.6.9. I am not suggesting this as an ideal solution but downgrading to 0.6.5 resolved this issue for us.
use the all() method
for user in User.objects.all():
print user.first_name
I am having issues getting mongodb working with Django, my setup is Mac OS X 10.7. I followed the tutorial available here: http://django-mongodb.org/topics/setup.html . I have tried with both virtualenv and without (this is my first time installing django so I shouldn't have any conflicts).
In settings.py
DATABASES = {
'default' : {
'ENGINE' : 'django_mongodb_engine',
'NAME' : 'my_database'
}
}
In firstapp.models.py
from django.db import models
from djangotoolbox.fields import ListField
class Post(models.Model):
title = models.CharField()
text = models.TextField()
tags = ListField()
comments = ListField()
In my shell (python manage.py shell), I try running:
from mydjango.firstapp.models import Post
post = Post.objects.create();
But I keep getting the following: DatabaseError: could not connect to localhost:27017: [Errno 61] Connection refused (full traceback)
If I switch settings.py to the following:
DATABASES = {
'default': {
'ENGINE': 'django_mongodb_engine',
'NAME': 'my_database',
'USER': '',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '27017',
'SUPPORTS_TRANSACTIONS': False,
},
}
I get this error in the shell: ImproperlyConfigured: port must be an instance of int
Did you setup MongoDB separately? The howto you link doesn't seem to go over the MongoDB configuration. It assumes the database is already running. In any case MongoDB seems down, or is at least listening somewhere else.
The last error ("...instance of int") is just because you specified '27017' (a string) instead of 27017 in the configuration dictionary. But even then it should be equivalent to the first, simpler configuration.
In case MongoDB is running but you still get this error while trying to connect from another machine, it may be due to Firewall running on the MongoDB server.
I was running into this exact same error on CentOS 6.5 running MongoDB 2.6.0. Disabling firewall on the machine resolved the issue for me.
If you are creating models at models.py then there is an example below
models.py
from mongoengine import *
class UserLocation(Document):
message = StringField(required=True, max_length=200)
You dont need the sqlite3 if you are not using it and only the mongodb then you have an optio to comment it
settings.py
# DATABASES = {
# 'default': {
# 'ENGINE': 'django.db.backends.sqlite3',
# 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
# }
# }
Now in
views.py
from .models import UserLocation
def save_function(request):
msg = "shinto"
# saving the data to the database
data = UserLocation(message = msg)
data.save()
# reading the data
read_data = json.loads(UserLocation.objects().to_json())
print read_data
There is also another method and its very simple (you don't need to create models in models.py)
views.py
from pymongo import MongoClient
client = MongoClient(port=27017)
db = client.testing_db # use a database called testing_db
collection = db.files # inside that db a collection called files
def a_new_function():
fooman = {"name" : "shinto", "age" : 25}
collection.insert(fooman)
# if you need to display only the name "shinto
#cursor = collection.find({"name" : "shinto"})
# if you need to display all then empty
cursor = collection.find({})
for document in cursor:
print(document)
could not connect to localhost:27017 Connection refused or you see 111 is because you haven't either installed or opened mongodb in another terminal
For installation on ubuntu do the following
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
if Ubuntu 12.04 (deprecated) then in terminal
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu precise/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list
if Ubuntu 14.04 then in terminal
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list
if Ubuntu 16.04 then in terminal
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list
After do
sudo apt-get update
sudo apt-get install -y mongodb-org
After installation then start mongodb
sudo service mongod start
To stop mongodb do
sudo service mongod stop
To restart mongodb
sudo service mongod restart
Finally you can use
mongo --host 127.0.0.1:27017
which will solve the issue
Thank You
From your question it seems like you are trying to use Django with Mongodb. In which case why do you need Mongoengine?
The official Mongodb documentation talks about djongo. It works by translating SQL queries into query documents.
You don't need Mongoengine to run it.
All native Django contrib modules (eg. admin, user, session) work without any modification.
MongoEngine requires rewriting Django modules and last I checked, the native admin module didn't run on MongoEngine.
Your existing models run without any ORM translation as well.