Django Elasticsearch AWS httplib UnicodeDecodeError - django

Question

I'm trying to setup Elasticsearch with Django (without Haystack).
Everything works perfectly locally.
But when I try to use the elasticsearch-py client with IAM based authentication on AWS I get this error :
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 827, in _send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position132: ordinal not in range(128)
I tried to use aws-es-connection and requests-aws4auth but I get the same error.
It only works when I allow open access on AWS and use only elasticsearch-py like so
from elasticsearch import Elasticsearch, RequestsHttpConnection
ES_CLIENT = Elasticsearch(
['search-domain-xxx.us-east-1.es.amazonaws.com'],
connection_class=RequestsHttpConnection
)
But I want something more secure ...
I think it is a utf-8/unicode/str problem but I can't manage to resolve it :(

Answer 1

I finally manage to fix the bug by using a custom serializer :
from elasticsearch import Elasticsearch, RequestsHttpConnection, serializer, compat, exceptions
class JSONSerializerPython2(serializer.JSONSerializer):
"""Override elasticsearch library serializer to ensure it encodes utf characters during json dump.
See original at: https://github.com/elastic/elasticsearch-py/blob/master/elasticsearch/serializer.py#L42
A description of how ensure_ascii encodes unicode characters to ensure they can be sent across the wire
as ascii can be found here: https://docs.python.org/2/library/json.html#basic-usage
"""
def dumps(self, data):
# don't serialize strings
if isinstance(data, compat.string_types):
return data
try:
return json.dumps(data, default=self.default, ensure_ascii=True)
except (ValueError, TypeError) as e:
raise exceptions.SerializationError(data, e)
and then pass the serializer to elasticsearch
from elasticsearch import Elasticsearch
es = Elasticsearch(..., serializer=JSONSerializerPython2())
I found the solution here

Answer 2

If someone can find the reason behind this error, I would be immensely grateful. Things I have tried with no luck:
encode things to 'utf-8' before writing (result: no errors thrown, es index not even created)
something like the code-snippet below (result: no errors thrown, es index not even created)
def convert_unicode_to_str(item):
if isinstance(item, basestring):
return str(item)
.....
.....
[Note: This is a bit of a hack, but it gets the job done]
So, sys.setdefaultencoding(..) is not available in the sys module's namespace. This is solved by calling the reload function on sys.
Now, prior to writing to ES, setting the default encoding to 'utf-8' solves this problem. Just to be sure that nothing else breaks as a result of this, I think it is a good idea to reset the default encoding to what it was before the switch.
reload(sys)
# keep track of what the old encoding was
old_encoding = sys.getdefaultencoding()
# set the default encoding to `itf-8`
sys.setdefaultencoding('utf-8')
#################
## write to es ##
#################
# reset the state of the world to what it was
sys.setdefaultencoding(old_encoding)

Answer 3

extending Debosmit Ray's solution for django-haystack.
add to search_indexes:
def set_default_utf8(method):
"""
Set utf-8 as default encoding, perform `method`,
then set the old encoding back.
"""
def wrapper(*args, **kwargs):
reload(sys)
old_encoding = sys.getdefaultencoding()
sys.setdefaultencoding('utf-8')
result = method(*args, **kwargs)
sys.setdefaultencoding(old_encoding)
return result
return wrapper
class EntryIndex(SearchIndex):
text = CharField(document=True, use_template=True)
...
#set_default_utf8
def update(self, using=None):
"""Overrides entire index update to use UTF-8."""
super(StreamEntryIndex, self).update(using)
#set_default_utf8
def update_object(self, instance, using=None, **kwargs):
"""Overrides index update for a single object using UTF-8."""
super(StreamEntryIndex, self).update_object(instance, using, **kwargs)