I have django 3.2.7, celery 5.2.1, redis 3.5.3
I have next celery settings. (REDIS_PASSWORD) is env variable:
CELERY_BROKER_URL = f'redis://:{REDIS_PASSWORD}#redis:6379/4'
CELERY_BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600}
CELERY_RESULT_BACKEND = f'redis://:{REDIS_PASSWORD}#redis:6379/1'
CELERY_ACCEPT_CONTENT = ['application/json']
But when I start my docker-compose app, it shows me
celery | --- ***** -----
celery | -- ******* ---- Linux-5.11.0-34-generic-x86_64-with-glibc2.31 2021-09-16 10:20:11
celery | - *** --- * ---
celery | - ** ---------- [config]
celery | - ** ---------- .> app: project:0x7f5cd0df0880
celery | - ** ---------- .> transport: redis://redis:6379/0 <==== NO CHANGES HERE
celery | - ** ---------- .> results: redis://:**#redis:6379/1
celery | - *** --- * --- .> concurrency: 16 (prefork)
celery | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
celery | --- ***** -----
How can I set it?
Problem was solved by removing environment params from docker-compose file.
Related
Not sure why it throws this key error.
My project/celery.py:
import os
from celery import Celery
# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
app = Celery('project')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print(f'Request: {self.request!r}')
my project/init.py:
from .celery import app as celery_app
__all__ = ('celery_app',)
My app/tasks.py
from celery import shared_task
from celery.schedules import crontab
from project.celery import app
#shared_task
def my_test():
print('Celery from task says Hi')
app.conf.beat_schedule = {
'my-task-every-2-minutes': {
'task': 'my_test',
'schedule': crontab(minute='*/1'),
},
}
when I run the command: celery -A project beat -l info
I can see the trigger every 1 min there
[2022-12-22 12:38:00,005: INFO/MainProcess] Scheduler: Sending due task my-task-every-2-minutes (celery_app.my_test)
when running celery -A project worker -l info
and triggers sends info I get KeyError: my_test
It seems that it cannot find the task with that key
but from my info everything running fine :
-------------- celery#192.168.1.131 v5.2.7 (dawn-chorus)
--- ***** -----
-- ******* ---- macOS-10.13.6-x86_64-i386-64bit 2022-12-22 12:43:22
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: project:0x10c757610
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. project.celery.debug_task
. user_statistic_status.tasks.my_test
I am building a web application that requires some long running tasks to be on AWS ECS using celery as a distributed task queue. The problem I am facing is that my celery worker running on ECS is not receiving tasks from SQS even though it seems to be connected to it.
Following are the logs from ECS task.
/usr/local/lib/python3.8/site-packages/celery/platforms.py:797: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
-------------- celery#ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal v5.0.1 (singularity)
--- ***** -----
-- ******* ---- Linux-4.14.252-195.483.amzn2.x86_64-x86_64-with-glibc2.2.5 2021-12-14 06:39:58
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: emptive_portal:0x7fbfda752310
- ** ---------- .> transport: sqs://XXXXXXXXXXXXXXXX:**#localhost//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> emptive-celery2.fifo exchange=sync(direct) key=emptive-celery.fifo
[tasks]
. import_export_celery.tasks.run_export_job
. import_export_celery.tasks.run_import_job
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
2021-12-14 06:39:58 [INFO] Connected to sqs://XXXXXXXXXXXXXXXXX:**#localhost//
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
2021-12-14 06:39:58 [INFO] celery#ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal ready.
To be noted, I have ran the same container, that I have deployed to ECS, locally on the same machine as the django webserver is on which is sending tasks. That celery worker doesn't have any problem receiving tasks.
I have also tried giving ecsTaskExecutionRole full permissions to SQS but that doesn't seem to affect anything. Any help would be appreciated.
EDIT: Forgot to show my celery broker config in django settings.py
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
# SQS CONFIG
CELERY_BROKER_URL = "sqs://%s:%s#" % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
BROKER_TRANSPORT_OPTIONS = {
'region': 'us-east-2',
'polling_interval': 20,
}
CELERY_RESULT_BACKEND = None
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_SEND_EVENTS = False
So I finally fixed this. The issue was really stupid on my part. :) I just had to replace BROKER_TRANSPORT_OPTIONS with CELERY_BROKER_TRANSPORT_OPTIONS in the celery config.
New config:
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
# SQS CONFIG
CELERY_BROKER_URL = "sqs://%s:%s#" % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_BROKER_TRANSPORT_OPTIONS = {
'region': 'us-east-2',
'polling_interval': 20,
}
CELERY_RESULT_BACKEND = None
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_SEND_EVENTS = False
I want to setup docker-compose for my app, which contains Django, Celery (+Beat), RabbitMQ and PSQL. My problem is that celery beat container does not work as intended (does not schedule a task with interval 10s).
Docker logs - celery beat wakes up in 5.00 minutes, celery worker works fine
celery-worker_1 | [2021-03-29 21:05:58,201: INFO/MainProcess] mingle: all alone
celery-worker_1 | [2021-03-29 21:05:58,225: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/fixups/django.py:205: UserWarning: Using settings.DEBUG leads to a memory
celery-worker_1 | leak, never use this setting in production environments!
celery-worker_1 | warnings.warn('''Using settings.DEBUG leads to a memory
celery-worker_1 | [2021-03-29 21:05:58,225: INFO/MainProcess] celery#247ea26ee17b ready.
celery-beat_1 | [2021-03-29 21:06:00,525: DEBUG/MainProcess] Setting default socket timeout to 30
celery-beat_1 | [2021-03-29 21:06:00,526: INFO/MainProcess] beat: Starting...
celery-beat_1 | [2021-03-29 21:06:00,749: DEBUG/MainProcess] Current schedule:
celery-beat_1 | <ScheduleEntry: celery.backend_cleanup celery.backend_cleanup() <crontab: 0 4 * * * (m/h/d/dM/MY)>
celery-beat_1 | [2021-03-29 21:06:00,749: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
celery-beat_1 | [2021-03-29 21:06:00,751: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
Dockerfile
FROM python:3
ENV PYTHONBUFFERED=1
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install -r requirements.txt
docker-compose.yml
version: '3.8'
services:
server:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/usr/src/app
ports:
- "8000:8000"
env_file:
- .env
depends_on:
- pgdb
- rabbit
- celery-worker
celery-worker:
build: .
command: sh -c 'sleep 8; celery -A core worker -l INFO'
env_file:
- .env
volumes:
- .:/usr/src/app
depends_on:
- pgdb
- rabbit
celery-beat:
build: .
command: sh -c 'sleep 10; celery -A core beat -l debug'
env_file:
- .env
volumes:
- .:/usr/src/app
depends_on:
- pgdb
- rabbit
- celery-worker
pgdb:
image: postgres
env_file:
- .env
volumes:
- pgdata:/var/lib/postgresql/data/
ports:
- "5432:5432"
restart: on-failure
rabbit:
image: "rabbitmq:3-management-alpine"
env_file:
- .env
ports:
- "5672:5672"
- "15672:15672"
restart: on-failure
volumes:
pgdata: {}
settings.py
CELERY_BROKER_URL = 'amqp://admin:mypass#rabbit:5672/'
CELERY_RESULT_BACKEND = 'db+postgresql://news_user:news#pgdb/news'
CELERY_BEAT_SCHEDULE = {
'scheduled_task': {
'task': 'news.tasks.get_news',
'schedule': 10.0
}
}
I'm trying to setup Celery to talk with AWS SQS. Celery put log as below and after that container is going down and starting up again in loop. Log as below:
trim...
[2020-08-05 09:32:35,715: DEBUG/MainProcess] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
[2020-08-05 09:32:35,798: DEBUG/MainProcess] Changing event name from before-call.apigateway to before-call.api-gateway
[2020-08-05 09:32:35,799: DEBUG/MainProcess] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
[2020-08-05 09:32:35,801: DEBUG/MainProcess] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
[2020-08-05 09:32:35,801: DEBUG/MainProcess] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
[2020-08-05 09:32:35,802: DEBUG/MainProcess] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
[2020-08-05 09:32:35,802: DEBUG/MainProcess] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
[2020-08-05 09:32:35,805: DEBUG/MainProcess] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
[2020-08-05 09:32:35,805: DEBUG/MainProcess] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
[2020-08-05 09:32:35,806: DEBUG/MainProcess] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
[2020-08-05 09:32:35,806: DEBUG/MainProcess] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
[2020-08-05 09:32:35,806: DEBUG/MainProcess] Setting config variable for region to {'eu-west-1'}
[2020-08-05 09:32:35,808: DEBUG/MainProcess] Loading JSON file: /usr/local/lib/python3.8/site-packages/botocore/data/endpoints.json
[2020-08-05 09:32:35,815: DEBUG/MainProcess] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fed0d144940>
[2020-08-05 09:32:35,899: DEBUG/MainProcess] Loading JSON file: /usr/local/lib/python3.8/site-packages/botocore/data/sqs/2012-11-05/service-2.json
[2020-08-05 09:32:35,902: DEBUG/MainProcess] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7fed096053a0>
[2020-08-05 09:32:36,907: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
-------------- celery#ip-10-10-12-215.eu-central-1.compute.internal v4.4.6 (cliffs)
--- ***** -----
-- ******* ---- Linux-4.14.177-139.254.amzn2.x86_64-x86_64-with-glibc2.2.5 2020-08-05 09:32:35
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: app_name:0x7fed0d6f18e0
- ** ---------- .> transport: sqs://aws_access_key:**#localhost//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. app_name.tasks.check_new_users
. app_name.tasks.send_mail_for_new_user
. app_name.tasks.test_task
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
Settings in Django are as follows: (variables are taken from SSM)
AWS_REGION = os.getenv("AWS_REGION")
BROKER_URL = os.getenv("BROKER_URL", "amqp://app_name-rabbitmq//")
BROKER_TRANSPORT = 'sqs'
BROKER_TRANSPORT_OPTIONS = {
"region": {AWS_REGION},
"polling_interval": 60,
"queue_name_prefix": "prod-"
}
CELERY_BROKER_URL = BROKER_URL
CELERY_BROKER_TRANSPORT_OPTIONS = BROKER_TRANSPORT_OPTIONS
CELERY_ACCEPT_CONTENT = ['json', 'yaml']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_SEND_EVENTS = False
CELERY_RESULT_BACKEND = None
RESULT_BACKEND = None
CELERY_IMPORTS = ("app_name.tasks",)
CELERYBEAT_SCHEDULE = {
"check_new_users": {"task": "tasks.app_name.check_new_users", "schedule": crontab(hour="9,17", minute=0,)},
}
Does any one had situation like that and could help?
IAM role is SQS full access for time being.
EDIT: If any other detail is needed please let me know.
I'm using Django with celery:
In celery.py I have:
app.conf.beat_schedule = {
'send_notes_email': {
'task': 'send_notes_email_async',
'schedule': crontab(minute=3),
},
}
The task is set:
#app.task(bind=True, name='send_notes_email_async', max_retries=3)
def send_notes_email_async():
print('a')
send_notes_email()
Celery is working, recognize the tasks, but doesn't trigger after 3 minutes.
Also no errors.
Redis reports keys
Celery tasks are working without schedule
-------------- celery#WIN-U5VDSO5V1CK v4.2.1 (windowlicker)
---- **** -----
--- * *** * -- Windows-7-6.1.7601-SP1
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: PH:0x3d2d128
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 2 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. send_notes_email_async
Your beat schedule configuration is wrong, minute=3 in cronjob means that execute the task at minute 3 (01:03, 02:03, 03:03, 04:03...).
This is what you want, which executes the task every 3 minutes:
app.conf.beat_schedule = {
'send_notes_email': {
'task': 'send_notes_email_async',
'schedule': crontab(minute=*/3),
},
}