Not sure why it throws this key error.
My project/celery.py:
import os
from celery import Celery
# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
app = Celery('project')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print(f'Request: {self.request!r}')
my project/init.py:
from .celery import app as celery_app
__all__ = ('celery_app',)
My app/tasks.py
from celery import shared_task
from celery.schedules import crontab
from project.celery import app
#shared_task
def my_test():
print('Celery from task says Hi')
app.conf.beat_schedule = {
'my-task-every-2-minutes': {
'task': 'my_test',
'schedule': crontab(minute='*/1'),
},
}
when I run the command: celery -A project beat -l info
I can see the trigger every 1 min there
[2022-12-22 12:38:00,005: INFO/MainProcess] Scheduler: Sending due task my-task-every-2-minutes (celery_app.my_test)
when running celery -A project worker -l info
and triggers sends info I get KeyError: my_test
It seems that it cannot find the task with that key
but from my info everything running fine :
-------------- celery#192.168.1.131 v5.2.7 (dawn-chorus)
--- ***** -----
-- ******* ---- macOS-10.13.6-x86_64-i386-64bit 2022-12-22 12:43:22
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: project:0x10c757610
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. project.celery.debug_task
. user_statistic_status.tasks.my_test
Related
I am building a web application that requires some long running tasks to be on AWS ECS using celery as a distributed task queue. The problem I am facing is that my celery worker running on ECS is not receiving tasks from SQS even though it seems to be connected to it.
Following are the logs from ECS task.
/usr/local/lib/python3.8/site-packages/celery/platforms.py:797: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
-------------- celery#ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal v5.0.1 (singularity)
--- ***** -----
-- ******* ---- Linux-4.14.252-195.483.amzn2.x86_64-x86_64-with-glibc2.2.5 2021-12-14 06:39:58
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: emptive_portal:0x7fbfda752310
- ** ---------- .> transport: sqs://XXXXXXXXXXXXXXXX:**#localhost//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> emptive-celery2.fifo exchange=sync(direct) key=emptive-celery.fifo
[tasks]
. import_export_celery.tasks.run_export_job
. import_export_celery.tasks.run_import_job
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
2021-12-14 06:39:58 [INFO] Connected to sqs://XXXXXXXXXXXXXXXXX:**#localhost//
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
2021-12-14 06:39:58 [INFO] celery#ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal ready.
To be noted, I have ran the same container, that I have deployed to ECS, locally on the same machine as the django webserver is on which is sending tasks. That celery worker doesn't have any problem receiving tasks.
I have also tried giving ecsTaskExecutionRole full permissions to SQS but that doesn't seem to affect anything. Any help would be appreciated.
EDIT: Forgot to show my celery broker config in django settings.py
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
# SQS CONFIG
CELERY_BROKER_URL = "sqs://%s:%s#" % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
BROKER_TRANSPORT_OPTIONS = {
'region': 'us-east-2',
'polling_interval': 20,
}
CELERY_RESULT_BACKEND = None
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_SEND_EVENTS = False
So I finally fixed this. The issue was really stupid on my part. :) I just had to replace BROKER_TRANSPORT_OPTIONS with CELERY_BROKER_TRANSPORT_OPTIONS in the celery config.
New config:
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
# SQS CONFIG
CELERY_BROKER_URL = "sqs://%s:%s#" % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_BROKER_TRANSPORT_OPTIONS = {
'region': 'us-east-2',
'polling_interval': 20,
}
CELERY_RESULT_BACKEND = None
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_SEND_EVENTS = False
I have django 3.2.7, celery 5.2.1, redis 3.5.3
I have next celery settings. (REDIS_PASSWORD) is env variable:
CELERY_BROKER_URL = f'redis://:{REDIS_PASSWORD}#redis:6379/4'
CELERY_BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600}
CELERY_RESULT_BACKEND = f'redis://:{REDIS_PASSWORD}#redis:6379/1'
CELERY_ACCEPT_CONTENT = ['application/json']
But when I start my docker-compose app, it shows me
celery | --- ***** -----
celery | -- ******* ---- Linux-5.11.0-34-generic-x86_64-with-glibc2.31 2021-09-16 10:20:11
celery | - *** --- * ---
celery | - ** ---------- [config]
celery | - ** ---------- .> app: project:0x7f5cd0df0880
celery | - ** ---------- .> transport: redis://redis:6379/0 <==== NO CHANGES HERE
celery | - ** ---------- .> results: redis://:**#redis:6379/1
celery | - *** --- * --- .> concurrency: 16 (prefork)
celery | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
celery | --- ***** -----
How can I set it?
Problem was solved by removing environment params from docker-compose file.
I'm trying to setup Celery to talk with AWS SQS. Celery put log as below and after that container is going down and starting up again in loop. Log as below:
trim...
[2020-08-05 09:32:35,715: DEBUG/MainProcess] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
[2020-08-05 09:32:35,798: DEBUG/MainProcess] Changing event name from before-call.apigateway to before-call.api-gateway
[2020-08-05 09:32:35,799: DEBUG/MainProcess] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
[2020-08-05 09:32:35,801: DEBUG/MainProcess] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
[2020-08-05 09:32:35,801: DEBUG/MainProcess] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
[2020-08-05 09:32:35,802: DEBUG/MainProcess] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
[2020-08-05 09:32:35,802: DEBUG/MainProcess] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
[2020-08-05 09:32:35,805: DEBUG/MainProcess] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
[2020-08-05 09:32:35,805: DEBUG/MainProcess] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
[2020-08-05 09:32:35,806: DEBUG/MainProcess] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
[2020-08-05 09:32:35,806: DEBUG/MainProcess] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
[2020-08-05 09:32:35,806: DEBUG/MainProcess] Setting config variable for region to {'eu-west-1'}
[2020-08-05 09:32:35,808: DEBUG/MainProcess] Loading JSON file: /usr/local/lib/python3.8/site-packages/botocore/data/endpoints.json
[2020-08-05 09:32:35,815: DEBUG/MainProcess] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fed0d144940>
[2020-08-05 09:32:35,899: DEBUG/MainProcess] Loading JSON file: /usr/local/lib/python3.8/site-packages/botocore/data/sqs/2012-11-05/service-2.json
[2020-08-05 09:32:35,902: DEBUG/MainProcess] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7fed096053a0>
[2020-08-05 09:32:36,907: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
-------------- celery#ip-10-10-12-215.eu-central-1.compute.internal v4.4.6 (cliffs)
--- ***** -----
-- ******* ---- Linux-4.14.177-139.254.amzn2.x86_64-x86_64-with-glibc2.2.5 2020-08-05 09:32:35
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: app_name:0x7fed0d6f18e0
- ** ---------- .> transport: sqs://aws_access_key:**#localhost//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. app_name.tasks.check_new_users
. app_name.tasks.send_mail_for_new_user
. app_name.tasks.test_task
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
Settings in Django are as follows: (variables are taken from SSM)
AWS_REGION = os.getenv("AWS_REGION")
BROKER_URL = os.getenv("BROKER_URL", "amqp://app_name-rabbitmq//")
BROKER_TRANSPORT = 'sqs'
BROKER_TRANSPORT_OPTIONS = {
"region": {AWS_REGION},
"polling_interval": 60,
"queue_name_prefix": "prod-"
}
CELERY_BROKER_URL = BROKER_URL
CELERY_BROKER_TRANSPORT_OPTIONS = BROKER_TRANSPORT_OPTIONS
CELERY_ACCEPT_CONTENT = ['json', 'yaml']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_SEND_EVENTS = False
CELERY_RESULT_BACKEND = None
RESULT_BACKEND = None
CELERY_IMPORTS = ("app_name.tasks",)
CELERYBEAT_SCHEDULE = {
"check_new_users": {"task": "tasks.app_name.check_new_users", "schedule": crontab(hour="9,17", minute=0,)},
}
Does any one had situation like that and could help?
IAM role is SQS full access for time being.
EDIT: If any other detail is needed please let me know.
project/project/settings.py
...
CELERY_BEAT_SCHEDULE = {
'find-subdomains': {
'task': 'subdiscovery.tasks.mytask',
'schedule': 10.0
}
}
project/subdiscovery/tasks.py
from __future__ import absolute_import, unicode_literals
from celery import shared_task
from subdiscovery.models import Domain
#shared_task
def mytask():
print(Domain.objects.all())
return 99
The celery worker shows an empty QuerySet:
celery_worker_1 | [2019-08-12 07:07:44,229: WARNING/ForkPoolWorker-2] <QuerySet []>
celery_worker_1 | [2019-08-12 07:07:44,229: INFO/ForkPoolWorker-2] Task subdiscovery.tasks.mytask[60c59024-cd19-4ce9-ae69-782a3a81351b] succeeded in 0.004897953000181587s: 99
However, importing the same model works in a python shell:
./manage.py shell
>>> from subdiscovery.models import Domain
>>> Domain.objects.all()
<QuerySet [<Domain: example1.com>, <Domain: example2.com>, <Domain: example3.com>]>
I should mention it's running in a docker stack
EDIT:
Ok, entering the running docker container
docker exec -it <web service container id> /bin/sh
and running
$ celery -A project worker -l info
works as expected:
[2019-08-13 05:12:28,945: INFO/MainProcess] Received task: subdiscovery.tasks.mytask[7b2760cf-1e7f-41f8-bc13-fa4042eedf33]
[2019-08-13 05:12:28,957: WARNING/ForkPoolWorker-8] <QuerySet [<Domain: uber.com>, <Domain: example1.com>, <Domain: example2.com>, <Domain: example3.com>]>
Here's what the docker-compose.yml looks like
version: '3'
services:
web:
build: .
image: app-image
ports:
- 80:8000
volumes:
- .:/app
command: gunicorn -b 0.0.0.0:8000 project.wsgi
redis:
image: "redis:alpine"
ports:
- 6379:6379
celery_worker:
working_dir: /app
command: sh -c './wait-for web:8000 && ./wait-for redis:6379 -- celery -A project worker -l info'
image: app-image
depends_on:
- web
- redis
celery_beat:
working_dir: /app
command: sh -c 'celery -A project beat -l info'
image: app-image
depends_on:
- celery_worker
Any idea why the worker started with docker-compose doesn't work, but entering the running container and starting a worker does?
Reposting from reddit https://www.reddit.com/r/docker/comments/cpoedr/different_behavior_when_starting_a_celery_worker/ewqx3mp?utm_source=share&utm_medium=web2x
Your problem here is that your celery worker doesn’t see sqlite database. You need to switch to different DB or make your ./app volume visible.
version: '3'
services:
...
celery_worker:
working_dir: /app
command: sh -c './wait-for web:8000 && ./wait-for redis:6379 -- celery -A project worker -l info'
image: app-image
volumes: # <-here
- .:/app
depends_on:
- web
- redis
...
I suggest switching to more production ready DB like postgres
I'm using Django with celery:
In celery.py I have:
app.conf.beat_schedule = {
'send_notes_email': {
'task': 'send_notes_email_async',
'schedule': crontab(minute=3),
},
}
The task is set:
#app.task(bind=True, name='send_notes_email_async', max_retries=3)
def send_notes_email_async():
print('a')
send_notes_email()
Celery is working, recognize the tasks, but doesn't trigger after 3 minutes.
Also no errors.
Redis reports keys
Celery tasks are working without schedule
-------------- celery#WIN-U5VDSO5V1CK v4.2.1 (windowlicker)
---- **** -----
--- * *** * -- Windows-7-6.1.7601-SP1
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: PH:0x3d2d128
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 2 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. send_notes_email_async
Your beat schedule configuration is wrong, minute=3 in cronjob means that execute the task at minute 3 (01:03, 02:03, 03:03, 04:03...).
This is what you want, which executes the task every 3 minutes:
app.conf.beat_schedule = {
'send_notes_email': {
'task': 'send_notes_email_async',
'schedule': crontab(minute=*/3),
},
}