Django GAE cron giving DeadlineExceededError - django

Question

I'm running Django 1.5 on GAE. I have a cron job that goes over several thousands of urls and grabs their "likes" count and saves it into DB. It can take easily more than 10 min to complete it. It works when I run it locally as a normal linux cron but fails with this error on GAE:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/wsgi.py", line 255, in __call__
response = self.get_response(request)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/base.py", line 175, in get_response
signals.got_request_exception.send(sender=self.__class__, request=request)
DeadlineExceededError
My setup:
app.yaml:
- url: /tasks/*
script: myproject.wsgi.application
login: admin
cron.yaml:
- description: update_facebook_resource
url: /tasks/update_facebook_resource
schedule: every day 04:05
timezone: Europe/Berlin
views.py
def update_facebook_resource(request):
resources = Resource.objects.filter(inactive=0).order_by('id')
url_start = "https://graph.facebook.com/fql?q=select++total_count+from+link_stat+where+url%3D"
url_end = "&access_token=..."
for item in resources:
url = item.link
url_final = url_start+ "%22" + url + "%22" + url_end
data = json.load(urllib2.urlopen(url_final))
likes = data["data"][0]["total_count"]
query = Resource.objects.get(id=item.id)
query.facebook_likes = likes
query.save(update_fields=['facebook_likes'])
return http.HttpResponse('ok')
what and how should I change so GAE lets me complete it? I've read this https://developers.google.com/appengine/articles/deadlineexceedederrors but it doesn't give me what I need really.
thanks

Answer 1

It's not a question of just getting GAE to let you complete the function. When developing for App Engine, you do need to think in a slightly different way, precisely because of things like the request deadline. In your case, you need to break the task up into chunks, and process each of those chunks individually.
You don't say if you're using django-nonrel with the GAE datastore, or if you're using Cloud SQL and therefore the standard Django API. If the former, you can use query cursors to keep track of your progress through the Resources. After each chunk, you can use deferred tasks to trigger the next chunk, passing it the cursor so it picks up where the last one left off.