I'm trying to offload a montecarlo task to Celery, and save the results to PostgreSQL AWS DB (RDS)
from views.py:
newResults = MonteCarloResultTrue.objects.create(htmlCompanyArray = "[]")
viewRun = runMonteCarloAsync.delay(checkedCompaniesIDs,newResults.id)
The object is created, but in tasks.py, the DB object is not being edited:
#app.task(bind=True)
def runMonteCarloAsync(self,checkedCompaniesIDs, calc_id):
newResults = MonteCarloResultTrue.objects.get(id=calc_id)
newResults.htmlCompanyArray = "[asdf]"
newResults.save()
How can I update the DB from the Celery task? Do I need to explicitly tell Celery where to look for the DB? (settings.py):
CELERY_accept_content = ['application/json']
CELERY_task_serializer = 'json'
CELERY_TASK_DEFAULT_QUEUE = 'django-queue-dev'
CELERY_BROKER_URL = 'sqs://{0}:{1}#'.format(
urllib.parse.quote(AWS_ACCESS_KEY_ID, safe=''),
urllib.parse.quote(AWS_SECRET_ACCESS_KEY, safe='')
)
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": "us-east-1",
'polling_interval': 20
}
CELERY_RESULT_BACKEND = 'django-db'
CELERY_CACHE_BACKEND = 'django-cache'
Procfile:
web: python manage.py runserver
celery_worker: celery -A rvm.settings.celery.app worker --loglevel=INFO
celery_beat: celery
What am I missing?
The problem was with my Procfile. I changed it to:
celery_worker: celery -A rvm worker --loglevel=INFO
and I can now save in the task (Using Celery 5.x)
Related
I'm running a Django app on AWS, with a PostgreSQL DB, using SQS. I'm trying to offload the montecarlo simulation onto Celery, but I am unable to save the results to my DB.
My view looks like:
runMonteCarloAsync.delay(checkedCompaniesIDs)
The task looks like:
#app.task(bind=True)
def runMonteCarloAsync(self,checkedCompaniesIDs):
# Do some montecarlo stuff
data = []
newResults = MonteCarloResultTrue(data=data)
newResults.save()
Here is my settings.py:
CELERY_accept_content = ['application/json']
CELERY_task_serializer = 'json'
CELERY_TASK_DEFAULT_QUEUE = 'django-queue-dev'
CELERY_BROKER_URL = 'sqs://{0}:{1}#'.format(
urllib.parse.quote(AWS_ACCESS_KEY_ID, safe=''),
urllib.parse.quote(AWS_SECRET_ACCESS_KEY, safe='')
)
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": "us-east-1",
'polling_interval': 20
}
CELERY_RESULT_BACKEND = 'django-db'
CELERY_CACHE_BACKEND = 'django-cache'
ProcFile:
web: python manage.py runserver
celery_worker: celery worker -A rvm.settings.celery.app --concurrency=1 --loglevel=INFO -n worker.%%h
celery_beat: celery
I can see the messages hitting SQS:
There are no new DB entires. I feel like I'm missing something, but I can't figure it out.
I'm using Django, Celery, and RabbitMQ for simple tasks on Ubuntu but celery gives no response.
I can't figure out why the task is pending in the browser, while it is done when I used the shell by executing python3 manage.py shell.
Here is my tasks.py file:
from celery import shared_task, task
#shared_task
def createContainer(container_data):
print(container_data,"create")
return "created"
#shared_task
def destroyContainer(container_data):
print(container_data,"destroy")
return "destroyed"
Here is my views.py file:
def post(self,request):
if str(request.data["process"]) == "create":
postdata = {
"image_name" : request.data["image_name"],
"image_tag" : request.data["image_tag"],
"owner" : request.user.id
}
# I tried to print the postdata variable before the task and it is working
createContainer.delay(postdata)
elif str(request.data["process"]) == "destroy":
postdata = {
"cont_id" : request.data["cont_id"]
}
# I tried to print the postdata variable before the task and it is working
destroyContainer.delay(postdata)
# I tried to print anything here, but it was not reachable and never executed
Here is the code I tried in the shell:
>>> from dockerapp.tasks import create_container
>>> create_container.delay("fake data")
>>> <AsyncResult: c37c47f3-6965-4f2e-afcd-01de60f82565>
Also, I can see the logs of celery here in another terminal by executing celery -A dockerproj worker -l info
It results in these lines when I used the shell:
Received task: dockerapp.tasks.create_container[c37c47f3-6965-4f2e-afcd-01de60f82565]
fake data #print
create #print
Task dockerapp.tasks.create_container[c37c47f3-6965-4f2e-afcd-01de60f82565] succeeded in 0.003456833990640007s
but it shows no results when I use it with the browser in a POST request.
However, I saw many solutions adding some lines to the settings.py file as celery configurations
I tried all of these lines:
CELERY_BROKER_URL = 'amqp://127.0.0.1'
CELERY_TIMEZONE = 'UTC'
CELERY_TRACK_STARTED = True
CELERY_TASK_TRACK_STARTED = True
CELERY_CACHE_BACKEND = 'amqp'
CELERY_TASK_TIME_LIMIT = 30 * 60
CELERY_IGNORE_RESULT = False
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']
I even tried this with celery terminal:
celery -A dockerproj worker -l info -P threads
celery -A dockerproj worker -l info --pool=solo (people said it fixed the issue in windows, however, i tried it)
i try to configure three queues/workers for celery in django.
settings.py
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/Berlin'
CELERY_QUEUES = (
Queue('manually_task', Exchange('manually_task'), routing_key='manually_task'),
Queue('periodically_task', Exchange('periodically_task'), routing_key='periodically_task'),
Queue('firsttime_task', Exchange('firsttime_task'), routing_key='firsttime_task'),
)
CELERY_ROUTES = {
'api.tasks.manually_task': {
'queue': 'manually_task',
'routing_key': 'manually_task',
},
'api.tasks.periodically_task': {
'queue': 'periodically_task',
'routing_key': 'periodically_task',
},
'api.tasks.firsttime_task': {
'queue': 'firsttime_task',
'routing_key': 'firsttime_task',
},
}
I have three tasks and every task should be have their own queue/worker.
My tasks look like this:
#shared_task
def manually_task(website_id):
print("manually_task");
website = Website.objects.get(pk=website_id)
x = Proxy(website, "49152")
x.startproxy()
x = None
#periodic_task(run_every=(crontab(hour=19, minute=15)), ignore_result=True)
def periodically_task():
websites = Website.objects.all()
for website in websites:
x = Proxy(website, "49153")
x.startproxy()
x = None
#shared_task
def firsttime_task(website_id):
website = Website.objects.get(pk=website_id)
x = Proxy(website, "49154")
x.startproxy()
x = None
Now for the first trial i start only one worker:
celery -A django-proj worker -Q manually_task -n manually_task
My problem is that the task not execute apparently, "manually_task" not printed.
Why its not working?
Based on the commments I suggest you should either provide queue name when you are calling a task from a view like manually_task.apply_async((webseite.pk,), queue='manually_task'), you or add default queue named celery when you start the worker as in celery -A django-proj worker -Q manually_task,celery
I have Django+Celery in Heroku, and Celery is set up as:
import djcelery
djcelery.setup_loader()
BROKER_URL = "django://" # tell kombu to use the Django database as the message queue
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
CELERY_ALWAYS_EAGER = False
CELERY_TIMEZONE = 'Europe/Madrid'
I have 2 tasks defined in tasks.py, one periodic and another that is executed on asynchronous calls:
#task
def test_one_shot():
print "One shot"
#periodic_task(run_every=crontab(minute="*/5"))
def test_periodic():
print "Periodic"
Heroku is configured with a main web worker and a auxiliar worker:
web: gunicorn config.wsgi:application ON
worker: python manage.py celery worker -B -l info ON
With this setup, I run the test_one_shot task as follows:
test_one_shot.apply_async(eta=datetime.now()+timedelta(minutes=2))
And although it appears as registered in the heroku logs:
Received task: test.tasks.test_one_shot[f29c609d-b6e8-45d4-808d-2ca690f029af] eta:[2016-08-07 00:09:30.262362+02:00]
It never executes. On the other hand, the periodic task test_periodic is executed as expected. What am I doing wrong?
Thanks!
EDIT: The task was executed was not appearing in the logs due a datetime time aware issue. However when the task is programmatically called, it is never executed.
I end up changing the celery backend to use RabbitMQ in Heroku following this guide, and the problem get solved.
Basically, I installed RabbitMQ on Heroku:
$ heroku addons:add cloudamqp
And set the new configuration for it:
import djcelery
djcelery.setup_loader()
CELERY_TIMEZONE = 'Europe/Madrid'
BROKER_URL = env("CLOUDAMQP_URL", default="django://")
BROKER_POOL_LIMIT = 1
BROKER_CONNECTION_MAX_RETRIES = None
CELERY_TASK_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ["json", "msgpack"]
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_ALWAYS_EAGER = False
if BROKER_URL == "django://":
INSTALLED_APPS += ("kombu.transport.django",)
I'm running Celery with Django and RabbitMQ and want to see the task states in the database table. Unfortunately no entries are written into the table djcelery_taskstate and I can't figure out why.
My settings:
CELERY_ENABLE_UTC = True
BROKER_URL = "amqp://guest:guest#localhost:5672/"
CELERY_RESULT_BACKEND = "database"
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_TRACK_STARTED = True
CELERY_SEND_EVENTS = True
CELERY_IMPORTS = ("project_management.tasks", "accounting.tasks", "time_tracking.tasks", )
CELERY_ALWAYS_EAGER = False
import djcelery
djcelery.setup_loader()
My Task:
class TestTask(Task):
def run(self, po_id):
self.update_state(state=states.STARTED, meta={'total': 0, 'done': False})
#do something..
self.update_state(state=states.SUCCESS, meta={'total': 100, 'done': True})
I'm starting the task as follows in a view:
TestTask.apply_async(args=[], kwargs={})
I'm starting celery workers as follows.
python manage.py celeryd -v 1 -B -s celery -E -l INFO
Console gives me the following output:
[2013-05-19 11:10:03,774: INFO/MainProcess] Task accounting.tasks.TestTask[5463b2ed-0eba-451d-b828-7a89fcd36348] succeeded in 0.0538640022278s: None
Any idea what is wrong with my setup?
You need to start up the snapshot camera as well in order to see the results in the database.
python manage.py celerycam
Once you have that running, you will be able to see entries in the djcelery tables.