Django - Return JSON, Error - django

Question

I got a View.py function that looks like this:
def GetAllCities(request):
cities = list(City.objects.all())
return HttpResponse(json.dumps(cities))
My City model looks like this
class City(models.Model):
city = models.CharField()
loc = models.CharField()
population = models.IntegerField()
state = models.CharField()
_id = models.CharField()
class MongoMeta:
db_table = "cities"
def __unicode__(self):
return self.city
I am using a MongoDB that looks like this
{
"_id" : ObjectId("5179837cbd7fe491c1f23227"),
"city" : "ACMAR",
"loc" : "[-86.51557, 33.584132]",
"state" : "AL",
"population" : 6055
}
I get the following error when trying to return the JSON from my GetAllCities function:
City ACMAR is not JSON serializable
So I tried this Instead:
def GetAllCities(request):
cities = serializers.serialize("json", City.objects.all())
return HttpResponse(cities)
And this works but It's very slow, it takes about 9 seconds(My database contains 30000 rows)
Should it take this long or am I doing something wrong?
I've built the same app in PHP, Rails and NodeJS.
In PHP it takes on average 2000ms, NodeJS = 800ms, Rails = 5882ms and Django 9395ms. Im trying to benchmark here so I wonder if there is a way to optimize my Django code or is this as fast as it gets?

Answer 1

For sure you do not need to return ALL cities, as you probably won't display all 30000 rows anyway (at least in user-friendly way). Consider a solution where you return only cities within some range from requested location. Mongo supports geospatial indexes, so there should be no problem in doing that. There are also many tutorials over the internet how to perform spatial filtering in Django/MongoDB.
def GetAllCities(request, lon, lat):
#Pseudo-code
cities = City.objects.filterWithingXkmFromLonLat(lon, lat).all()
cities = serializers.serialize("json", cities)
return HttpResponse(cities)
If you really, really need all cities, consider caching the response. Location, name and population of cities are not things which change dynamically, in a matter of let's say seconds. Cache the result and recalculate only every hour, day or more. Django supports cache out of the box
#cache_page(60 * 60)
def GetAllCities(request):
(...)

Answer 2

Another thing you can try to get a little more of speed is to get from db just the values you need and get the QuerySet to build the dictionary.
A simple query like this would work:
City.objects.all().values('id', 'city', 'loc', 'population', 'state')
Or you can put it in a manager:
class CitiesManager(models.Manager):
class as_dict(self):
return self.all().values('id', 'city', 'loc', 'population', 'state')
class City(models.Model):
.... your fields here...
objects = CitiesManager()
And then use it in your view as:
City.objects.as_dict()

Answer 3

FOUND A SOLUTION
I am benchmarking with different methods, one method is to see how fast one language/framework is to select ALL rows in a database and return it as JSON. I found a solution now that speeds it up by half the time!
My new views.py
def GetAllCities(request):
dictionaries = [ obj.as_dict() for obj in City.objects.all() ]
return HttpResponse(json.dumps({"Cities": dictionaries}), content_type='application/json')
And my new model
class City(models.Model):
city = models.CharField()
loc = models.CharField()
population = models.IntegerField()
state = models.CharField()
_id = models.CharField()
def as_dict(self):
return {
"id": self.id,
"city": self.city,
"loc": self.loc,
"population": self.population,
"state": self.state
# other stuff
}
class MongoMeta:
db_table = "cities"
def __unicode__(self):
return self.city
Found the solution here