Using UTF-8 encoded JSON fixture file in Django - django

Question

I'm trying to write a JSON initial data fixture that will be loaded after every call to syncdb.
I placed an initial_data.json file in my mysite/myapp/fixtures directory:
[
{
"model": "myapp.Person",
"pk": 1,
"fields": {
"first_name": "Tom",
"last_name": "Yam"
}
}
]
Everything is working when the file is encoded in ASCII, but when I save it in UTF-8 encoding (I need to use non-ASCII characters) I get to following error:
Problem installing fixture 'initial_data.json': Traceback (most recent call last):
File "D:\Tom\DjangoEnv\Lib\site-packages\django\core\management\commands\loaddata.py", line 190, in handle
for obj in objects:
File "D:\Tom\DjangoEnv\Lib\site-packages\django\core\serializers\json.py", line 47, in Deserializer
raise DeserializationError(e)
DeserializationError: No JSON object could be decoded
According to the Django documentation, I need to set ensure_ascii=False when working with non-ASCII data and JSON serializers, but I can't figure how to do it (since its being called from the syncdb function.
Any ideas how to use a UTF-8 encoded JASON file as a fixture?

Answer 1

load_data would not pass ensure_ascii option to serializer so you have two options:
convert data to ascii unicode escaped before loading it, ie:
import codecs
encoded = codecs.open('/tmp/tst.txt', 'r', 'utf-8').read().encode(
'ascii', 'backslashreplace')
open('/tmp/tst-encoded.txt', 'w').write(encoded)
write your own management command that would pass ensure_ascii
hope this helps.