Bulk index django model to elastic search - django

Question

I am trying to bulk index my django model to elastic search 6, my plan is to run this as a cron once a day to update the index.
import requests
data = serialize('json', CapitalSheet.objects.all())
data += "\n"
r = requests.post("http://127.0.0.1:9200/capitalsheet/_bulk", json = data)
print(r.content)
I am getting this error:
b'{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"},"status":400}'
If you can suggest something better, I would be glad.

Answer 1

I would recommend looking at the python library provided by elasticsearch. It will make it easier to do the bulk inserts. Here is a link to the docs:
https://elasticsearch-py.readthedocs.io/en/master/helpers.html#bulk-helpers
If you want to do it manually though, the ES bulk API actually requires two lines for every record you want to insert. The first line details what index and the type of operation, and the second line is the record to insert. For example your request body would look something like this:
{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value2" }
The ES documentation explains this well here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html