Just installed sqs with django and celery to do background tasks like sending notifications and firebase messages
I was using redis and it was fast when installed sqs the code is not async anymore, the errors even appear in the server terminal not the celery one!
here's my settings.py
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": "us-east-1",
}
CELERY_BROKER_TRANSPORT = 'sqs'
CELERY_BROKER_URL = f"sqs://{AWS_ACCESS_KEY_ID}:{AWS_SECRET_ACCESS_KEY}#"
CELERY_TASK_DEFAULT_QUEUE = "hikeapp"
CELERY_RESULT_BACKEND = None
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
when moved back to redis it became fast again, what am I doing wrong?
Related
I'm running a Django app on AWS, with a PostgreSQL DB, using SQS. I'm trying to offload the montecarlo simulation onto Celery, but I am unable to save the results to my DB.
My view looks like:
runMonteCarloAsync.delay(checkedCompaniesIDs)
The task looks like:
#app.task(bind=True)
def runMonteCarloAsync(self,checkedCompaniesIDs):
# Do some montecarlo stuff
data = []
newResults = MonteCarloResultTrue(data=data)
newResults.save()
Here is my settings.py:
CELERY_accept_content = ['application/json']
CELERY_task_serializer = 'json'
CELERY_TASK_DEFAULT_QUEUE = 'django-queue-dev'
CELERY_BROKER_URL = 'sqs://{0}:{1}#'.format(
urllib.parse.quote(AWS_ACCESS_KEY_ID, safe=''),
urllib.parse.quote(AWS_SECRET_ACCESS_KEY, safe='')
)
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": "us-east-1",
'polling_interval': 20
}
CELERY_RESULT_BACKEND = 'django-db'
CELERY_CACHE_BACKEND = 'django-cache'
ProcFile:
web: python manage.py runserver
celery_worker: celery worker -A rvm.settings.celery.app --concurrency=1 --loglevel=INFO -n worker.%%h
celery_beat: celery
I can see the messages hitting SQS:
There are no new DB entires. I feel like I'm missing something, but I can't figure it out.
i try to configure three queues/workers for celery in django.
settings.py
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/Berlin'
CELERY_QUEUES = (
Queue('manually_task', Exchange('manually_task'), routing_key='manually_task'),
Queue('periodically_task', Exchange('periodically_task'), routing_key='periodically_task'),
Queue('firsttime_task', Exchange('firsttime_task'), routing_key='firsttime_task'),
)
CELERY_ROUTES = {
'api.tasks.manually_task': {
'queue': 'manually_task',
'routing_key': 'manually_task',
},
'api.tasks.periodically_task': {
'queue': 'periodically_task',
'routing_key': 'periodically_task',
},
'api.tasks.firsttime_task': {
'queue': 'firsttime_task',
'routing_key': 'firsttime_task',
},
}
I have three tasks and every task should be have their own queue/worker.
My tasks look like this:
#shared_task
def manually_task(website_id):
print("manually_task");
website = Website.objects.get(pk=website_id)
x = Proxy(website, "49152")
x.startproxy()
x = None
#periodic_task(run_every=(crontab(hour=19, minute=15)), ignore_result=True)
def periodically_task():
websites = Website.objects.all()
for website in websites:
x = Proxy(website, "49153")
x.startproxy()
x = None
#shared_task
def firsttime_task(website_id):
website = Website.objects.get(pk=website_id)
x = Proxy(website, "49154")
x.startproxy()
x = None
Now for the first trial i start only one worker:
celery -A django-proj worker -Q manually_task -n manually_task
My problem is that the task not execute apparently, "manually_task" not printed.
Why its not working?
Based on the commments I suggest you should either provide queue name when you are calling a task from a view like manually_task.apply_async((webseite.pk,), queue='manually_task'), you or add default queue named celery when you start the worker as in celery -A django-proj worker -Q manually_task,celery
Not duplicate of Celery: No Result Backend Configured? because SQS is used.
Keep getting the following error:
No result backend is configured. Please see the documentation for more
information.
My production settings are the following:
CELERY_BROKER_URL = 'sqs://%s:%s#' % (
urllib.parse.quote(env.str('TASK_QUEUE_USER_ID'), safe=''),
urllib.parse.quote(env.str('TASK_QUEUE_USER_SECRET'), safe=''))
BROKER_URL = CELERY_BROKER_URL
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_RESULT_BACKEND = None # Disabling the results backend
RESULT_BACKEND = None # Disabling the results backend
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_DEFAULT_QUEUE = 'async_tasks'
SQS_QUEUE_NAME = 'async_tasks'
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_SEND_EVENTS = False
CELERY_BROKER_TRANSPORT_OPTIONS = {
'region': 'eu-west-2',
'polling_interval': 3,
'visibility_timeout': 3600,
}
CELERY_SEND_TASK_ERROR_EMAILS = True
#
# https://stackoverflow.com/questions/8048556/celery-with-amazon-sqs#8567665
#
CELERY_BROKER_TRANSPORT = 'sqs'
BROKER_TRANSPORT = 'sqs'
Running celery from the command line:
DJANGO_ENV=production celery -A async_tasks worker -l info
connects to SQS and polls, but when I try to do a demo call from the command line DJANGO_ENV=production python manage.py check_async:
from django.core.management.base import BaseCommand, CommandError
import async_tasks.tasks as tasks
class Command(BaseCommand):
help = 'Check if infrastructure for async tasks has been setup correctly.'
def handle(self, *args, **options):
try:
print('Sending async request.')
t = tasks.add.apply_async((2, 4))
out = t.get(timeout=1)
print(out)
print(t.status)
except Exception as e:
print(e)
raise CommandError('Error occured')
I get the error above. Have tried in development machine with redis and everything works well.
Any ideas?
You need a Celery Result Backend configured to be able to store and collect task results. Using Celery with an SQS broker w/o a result backend is ok for "fire and forget" patterns, but it's not enough if you want to be able to access the results of your tasks through methods like get().
Maybe this will help someone. The answer above is correct, but if you still want to use Django, SQS and Celery and still want to see the results you can use Django's ORM or Cache Framework as a backend by using the django-celery-results library.
Django-celery-results
Celery Documentation - ORM Cache Framework
I have Django+Celery in Heroku, and Celery is set up as:
import djcelery
djcelery.setup_loader()
BROKER_URL = "django://" # tell kombu to use the Django database as the message queue
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
CELERY_ALWAYS_EAGER = False
CELERY_TIMEZONE = 'Europe/Madrid'
I have 2 tasks defined in tasks.py, one periodic and another that is executed on asynchronous calls:
#task
def test_one_shot():
print "One shot"
#periodic_task(run_every=crontab(minute="*/5"))
def test_periodic():
print "Periodic"
Heroku is configured with a main web worker and a auxiliar worker:
web: gunicorn config.wsgi:application ON
worker: python manage.py celery worker -B -l info ON
With this setup, I run the test_one_shot task as follows:
test_one_shot.apply_async(eta=datetime.now()+timedelta(minutes=2))
And although it appears as registered in the heroku logs:
Received task: test.tasks.test_one_shot[f29c609d-b6e8-45d4-808d-2ca690f029af] eta:[2016-08-07 00:09:30.262362+02:00]
It never executes. On the other hand, the periodic task test_periodic is executed as expected. What am I doing wrong?
Thanks!
EDIT: The task was executed was not appearing in the logs due a datetime time aware issue. However when the task is programmatically called, it is never executed.
I end up changing the celery backend to use RabbitMQ in Heroku following this guide, and the problem get solved.
Basically, I installed RabbitMQ on Heroku:
$ heroku addons:add cloudamqp
And set the new configuration for it:
import djcelery
djcelery.setup_loader()
CELERY_TIMEZONE = 'Europe/Madrid'
BROKER_URL = env("CLOUDAMQP_URL", default="django://")
BROKER_POOL_LIMIT = 1
BROKER_CONNECTION_MAX_RETRIES = None
CELERY_TASK_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ["json", "msgpack"]
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_ALWAYS_EAGER = False
if BROKER_URL == "django://":
INSTALLED_APPS += ("kombu.transport.django",)
I have used celery for my Django application computation intensive tasks. I am using Redis as a broker, and as Celery backend result.
Everything works fine on localhost. But on Aws, AsyncResult.ready() always returns false. Seems like it does not maintain state of the task.
I have AWS Elasticache redis clusters, for Redis. And elasticbeanstalk for my Django app. Here is a snippet of my code:
ares = AsyncResult(task_id)
if(ares.ready()):
return HttpResponse(ares.get())
else:
return HttpResponse(status=202)
This 'ares.ready()' always returns false on AWS, but works fine on localhost, with local redis-server
Here is my celery Configuration in settings.py:
BROKER_URL = 'redis://interacta001.a4zxsx.0001.use1.cache.amazonaws.com:6379'
CELERY_RESULT_BACKEND = 'redis://interacta-001.a4zxsx.0001.use1.cache.amazonaws.com:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_IMPORTS = ("ebdjango.tasks", )
Note: The redis-server on clusters is working fine, and when ping from AMI (Amazon machine) using SSH, it responds quickly. Also I am using djcelery in installed apps.