Django + Heroku celery module not found - django

I'm trying to push my lastest version of my webapp live to Heroku. I'm pretty comfortable pushing to Heroku and this just doesn't seem right. For some reason I feel like Heroku is skipping my requirements.txt. Just in case I manually installed celery on my Heroku app.
I'm getting this specific problem with celery, but if Heroku is skipping my requirements.txt this might be a bigger problem.
1. If I run:
heroku run pip install celery
This let's me install the package over and over, shouldn't it kick back a "requirement already met" error?
2. When I try to push to heroku, I keep getting a
File "/app/config/_celery.py", line 4, in <module>
from celery import Celery
ModuleNotFoundError: No module named 'celery'
for the life of me I can't figure out why, I've uninstalled celery, reinstalled it locally. It's on my requirements.txt (Heroku should install it upon push to the remote). celery also seems to work locally just fine.
I'll include what I think is necessary, but let me know if I'm missing something that might provide the answer.
Here's my projects file structure:
POTRTMS(overall project folder)
|
+-config(holds settings)
| |
| +--settings
| | |
| | __init__.py
| | production.py
| | local.py
| | base.py
| |
| _celery.py
| __init__.py (this is the __init__.py i'm referencing below)
_celery.py
from __future__ import absolute_import, unicode_literals
import sys
import os
from celery import Celery
from celery.schedules import crontab
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.production')
app = Celery('POTRTMS')
# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
app.conf.beat_schedule = {
'insurance_date': {
'task': 'insurance_date',
'schedule': crontab(minute=0, hour=8),
},
}
__init.py__
from __future__ import absolute_import
from ._celery import app as celery_app

I was able to figure out the problem. Inside of my project root folder I had Pipfile and Pipfile.lock which sounds like a new way to do the requirements.txt.
I'm not sure how these files got into my project but after deleting them all is working.

Related

Celery - Received unregistered task of type 'core.tasks.scrape_dev_to'

Trying to get a celery-based scraper up and running. The celery worker seems to function on its own, but when I also run the celery beat server, the worker gives me this keyerror.
File "c:\users\myusername\.virtualenvs\django-news-scraper-dbqk-dk5\lib\site-packages\celery\worker\consumer\consumer.py", line 555, in on_task_received
strategy = strategies[type_]
KeyError: 'core.tasks.scrape_dev_to'
[2020-10-04 16:51:41,231: ERROR/MainProcess] Received unregistered task of type 'core.tasks.scrape_dev_to'.
The message has been ignored and discarded.
I've been through many similar answers on stackoverflow, but none solved my problem. I'll list things I tried at the end.
Project structure:
core -tasks
newsscraper -celery.py -settings.py
tasks:
import time
from newsscraper.celery import shared_task, task
from .scrapers import scrape
#task
def scrape_dev_to():
URL = "https://dev.to/search?q=django"
scrape(URL)
return
settings.py:
INSTALLED_APPS = [
'django.contrib.admin',
...
'django_celery_beat',
'core',
]
...
# I Added this setting while troubleshooting, got a new ModuleNotFound error for core.tasks
#CELERY_IMPORTS = (
# 'core.tasks',
#)
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_BEAT_SCHEDULE = {
"ScrapeStuff": {
'task': 'core.tasks.scrape_dev_to',
'schedule': 10 # crontab(minute="*/30")
}
}
celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'newsscraper.settings')
app = Celery('newsscraper')
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
When I run debug for the celery worker, I see that celery doesn't have the task I want (scrape_dev_to) registered. Shouldn't the app.autodiscover_tasks() call in celery.py take care of this? Here's the output:
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
I also get a ModuleNotFoundError when I try to add core.tasks to a CELERY_IMPORTS setting. This is my best guess for where the problem is, but I don't know how to solve it.
Things I tried:
Add core.tasks to a celery_imports setting. This causes a new error when I try to run the celery beat: ‘no module named ‘core.tasks’ ‘.
Hardcoding the name in the task: name='core.tasks.scrape_dev_to'
Specified the celery config explicitly when calling the worker: celery -A newsscraper worker -l INFO -settings=celeryconfig
Playing with my imports (from newsscraper.celery instead of from celery, for instance)
Adding some config code to the init.py for the module containing tasks (already had it in the init.py for module containing settings and celery.py)
Python manage.py check identified no issues
Calling the work with core.tasks explicitly: celery -A core.tasks worker -l INFO
I had the same problem and this setup solved it for me.
in your settings
CELERY_IMPORTS = [
'app_name.tasks',
]
and
# app_name/tasks.py
from celery import shared_task
#shared_task
def my_task(*args, **kwargs):
pass
Docs ref for imports.
This can occur when you configured a celery task and then removed it.
Just deconfigure the tasks and configure again
$ celery -A proj purge
or
from proj.celery import app
app.control.purge()
In settings.py, I have added the below line:
CELERY_IMPORTS = [
'app_name.tasks',]
and it worked for me.

Error: Unable to load celery application. The module tasks was not found

I'm trying to get celery up and running on Heroku as per instructions here
When I try to run "heroku local" however it fives me the following error:
10:05:42 PM worker.1 | Error:
10:05:42 PM worker.1 | Unable to load celery application.
10:05:42 PM worker.1 | The module tasks was not found.
Any help is much appreciated.
EDIT: It should be noted that I have a module tasks.py in my root directory with the following code in it:
import celery
app = celery.Celery('example')
#app.task
def add(x, y):
return x + y
Based on the comments, I think you need to populate the init.py in your project folder (same folder as celery.py). You can follow Celery official documentation.
This is what you should add to __init__.py:
from __future__ import absolute_import, unicode_literals
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
__all__ = ('celery_app',)
Hope this helps.

Keyerror from celery task delay

Following the celery getting started with Django instructions, I am able to run tasks, but not run the same task asynchronously using delay().
I added the following requirements to my Django project:
celery==4.3.0
redis==3.3.11
django-celery-results==1.1.2
psycopg2==2.7.3.1
django-cors-headers~=3.1.0
Created this celery.py in the pop_model project directory:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'pop_model.settings.local')
app = Celery('pop_model')
# namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
Inserted this code in the project init.py:
from __future__ import absolute_import, unicode_literals
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
__all__ = ('celery_app',)
Configured cors in the project settings and added these settings:
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'django-db' # defined in django_celery_results
CELERY_ACCEPT_CONTENT = ['json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
I can start redis, then run celery using these commands:
export DJANGO_SETTINGS_MODULE=pop_model.settings.local
celery worker -A pop_model --loglevel=info
In a python3 shell, I get these results:
>>> from pop_model.celery import debug_task
>>> debug_task()
Request: <Context: {'args': (), 'kwargs': {}}>
>>> task=debug_task.delay()
Traceback (most recent call last):
File "/Users/janee/.virtualenvs/pop-model/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
return obj.__dict__[self.__name__]
KeyError: 'backend'
I don't know how to resolve the missing backend key as CELERY_RESULT_BACKEND is defined in the settings file.
The only difference between a normal Python shell and manage.py shell is that it exports your settings module (project_name.settings) in the DJANGO_SETTINGS_MODULE environment variable.
If you run the same interpreter with the proper environment variable you should see no change. Then, it may be that your pop_model.settings.local path is not returning a proper settings module for your app to latch on, or you're using a modified manage.py script (for development environment separation, I suppose) where the settings module is properly loaded.
You should be able to call your function using
./manage.py shell
from your project directory, using the same intepreter of your virtual environment. This could also work because the DJANGO_SETTINGS_MODULE needs a path that is present in the interpreter's search path (more on that here) and you could be calling the interpreter from another directory.

Celery doesn't autodiscover tasks from all apps

I have a problem with autodiscovery tasks from all y apps.
For project
proj
|-- settings.py
|-- app_tasks_found
| |-- tasks.py
|
|-- app_cant_find_tasks
| |-- tasks.py
And following settings
INSTALLED_APPS = [
'proj.app_tasks_found',
'proj.app_cant_find_tasks',
]
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
from django.conf import settings
app = Celery('proj')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
After I start celery worker
celery worker -l info -A proj
It shows me only tasks from app_tasks_found, but not from app_cant_find_tasks
And of course celery raise error when I try to call task app_cant_find_tasks.tasks.test_task.delay()
Celery can find tasks from app_cant_find_tasks if I explicitly set CELERY_IMPORTS = ("proj.app_cant_find_tasks.tasks",)
These apps totally similar for me. I don't understand why celery can autodiscover tasks only from one app.
Question is, where should I look at to fix my problem?
I had a similar issue and in my case the problem was I was missing the init.py file in one of the apps.

Django Celery Scheduling a manage.py command

I need to update the solr index on a schedule with the command:
(env)$ ./manage.py update_index
I've looked through the Celery docs and found info on scheduling, but haven't been able to find a way to run a django management command on a schedule and inside a virtualenv. Would this be better run on a normal cron? And if so how would I run it inside the virtualenv? Anyone have experience with this?
Thanks for the help!
Django Celery Task Scheduling
project structure
[appname]/
├── [appname]/
│ ├── __init__.py
│ ├── settings.py
│ ├── urls.py
│ ├── celery.py
│ └── wsgi.py
├── [project1]/
│ ├── __init__.py
│ ├── tasks.py
│
└── manage.py
add below configuration in settings.py file:
STATIC_URL = '/static/'
BROKER_URL = 'redis://localhost:6379/0'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_RESULT_BACKEND = 'redis'
from celery.schedules import crontab
CELERY_TIMEZONE = 'UTC'
celery.py : holds celery task scheduler
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
import django
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'appname.settings')
from django.conf import settings
app = Celery('appname')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
**#scheduler**
app.conf.beat_schedule = {
'add-every-30-seconds': {
'task': 'project1.tasks.cleanup',
'schedule': 30.0,
'args': ()
},
}
app.conf.timezone = 'UTC'
init.py
from __future__ import absolute_import, unicode_literals
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
__all__ = ['celery_app']
tasks.py from project1
from celery import shared_task
import celery
import time
from django.core import management
#celery.task#(name='cleanup')
def cleanup():
try:
print ("in celery module")
"""Cleanup expired sessions by using Django management command."""
management.call_command("clearsessions", verbosity=0)
#PUT MANAGEMENT COMMAND HERE
return "success"
except:
print(e)
Task will run after every 30 seconds
Requirement fro windows:
redis server should be running
celery worker and celery beat should be running
run each below command on different terminal
celery -A appname worker -l info
celery -A appname beat -l info
Requirement fro Linux:
redis server should be running
celery worker and celery beat should be running
celery beat and worker can be started on same server
celery -A appname worker -l info -B
#tzenderman please let me know if I missed something.
For me this is working fine
To run your command periodically from a cron job, just wrap the command in a bash script that loads the virtualenv. For example, here is what we do to run manage.py commands:
django_cmd.sh:
#!/bin/bash
cd /var/www/website/
source venv/bin/activate
/var/www/website/manage.py $1 --settings=$2
Crontab:
MAILTO=webmaster#website.com
SETTINGSMODULE=website.settings_prod
5 * * * * /var/www/website/django_cmd.sh update_index $SETTINGSMODULE >> /dev/null
0 10 * * * /var/www/website/django_cmd.sh update_accounts $SETTINGSMODULE
I actually found a nice way of doing this using fabric + celery and I'm working on it now:
In app/tasks.py, create a fabric function with the manage.py commands you need, then decorate it with #periodic_task, add it to your celery schedule and it should be good to go.
UPDATE: I wasn't able to actually use Fabric + Celery because using fabric in the module caused it be recognized as a fabric file and the celery calls in the file didn't work.