celery error :'module' object has no attribute 'celery' - django

I try to start celery using command below:
celery -A converter worker --loglevel=info
but it doesn't work.
my converter.py is:
from __future__ import absolute_import, unicode_literals
from celery.task import task
#task
def ffmpeg_convert(input_file, bitrate):
#do something
and my celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings.base')
app = Celery('converter')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.conf.broker_url = 'redis://localhost:6379/0'
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
but I get following error:
Traceback (most recent call last):
File "/usr/local/bin/celery", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 14, in main
_main()
File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 326, in main
cmd.execute_from_commandline(argv)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 488, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 279, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 481, in setup_app_from_commandline
self.app = self.find_app(app)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 503, in find_app
return find_app(app, symbol_by_name=self.symbol_by_name)
File "/usr/local/lib/python2.7/dist-packages/celery/app/utils.py", line 366, in find_app
found = sym.celery
AttributeError: 'module' object has no attribute 'celery'
Do anyone has any idea?Thank for help

I've been experiencing a similar problem and I think the issue might be from where you're running celery -A proj worker --loglevel=info.
If you have a project structure like
directory
--virtualenv/
--proj/
----manage.py
----requirements.txt
----proj/
------settings.py
------celery.py
You need to run the command from --proj/

Looks like your celery.py file shadows the celery package.

Related

Django AppRegistryNotReady Exception after import from django.contrib.auth.models

Here is my folder structure:
.
├── app
├── project
├── scripts
└── tests
├── fuctional -> test_views.py
└── unit
Here is my test_script.py
from django.test import TestCase
from django.urls import reverse
from django.contrib.auth.models import User
from django.contrib.auth.hashers import make_password
from rest_framework.test import APIClient
from mrx.views import LoginView, LogoutView
When I launch python -m unittest tests/fuctional/test_views.py I get this error:
Traceback (most recent call last):
File "/home/edx/.pyenv/versions/3.8.1/lib/python3.8/runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/edx/.pyenv/versions/3.8.1/lib/python3.8/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/edx/.pyenv/versions/3.8.1/lib/python3.8/unittest/__main__.py", line 18, in <module>
main(module=None)
File "/home/edx/.pyenv/versions/3.8.1/lib/python3.8/unittest/main.py", line 100, in __init__
self.parseArgs(argv)
File "/home/edx/.pyenv/versions/3.8.1/lib/python3.8/unittest/main.py", line 147, in parseArgs
self.createTests()
File "/home/edx/.pyenv/versions/3.8.1/lib/python3.8/unittest/main.py", line 158, in createTests
self.test = self.testLoader.loadTestsFromNames(self.testNames,
File "/home/edx/.pyenv/versions/3.8.1/lib/python3.8/unittest/loader.py", line 220, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/home/edx/.pyenv/versions/3.8.1/lib/python3.8/unittest/loader.py", line 220, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "/home/edx/.pyenv/versions/3.8.1/lib/python3.8/unittest/loader.py", line 154, in loadTestsFromName
module = __import__(module_name)
File "/home/edx/PycharmProjects/mrx_3/tests/fuctional/test_views.py", line 3, in <module>
from django.contrib.auth.models import User
File "/home/edx/.pyenv/versions/MRX/lib/python3.8/site-packages/django/contrib/auth/models.py", line 2, in <module>
from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
File "/home/edx/.pyenv/versions/MRX/lib/python3.8/site-packages/django/contrib/auth/base_user.py", line 48, in <module>
class AbstractBaseUser(models.Model):
File "/home/edx/.pyenv/versions/MRX/lib/python3.8/site-packages/django/db/models/base.py", line 108, in __new__
app_config = apps.get_containing_app_config(module)
File "/home/edx/.pyenv/versions/MRX/lib/python3.8/site-packages/django/apps/registry.py", line 253, in get_containing_app_config
self.check_apps_ready()
File "/home/edx/.pyenv/versions/MRX/lib/python3.8/site-packages/django/apps/registry.py", line 136, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
I've tried with import Django and then django.setup() and it works but:
why I can't import from django.contrib.auth.models import User ?
How do you test outside a Django app code inside a Django app?
Edit:
I've also tried with pytest, using init.py with import Django and then django.setup() but I still got an error:
(MRX) edx#edx-VirtualBox:~/PycharmProjects/mrx_3$ pytest tests/fuctional/test_views.py
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.8.1, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/edx/PycharmProjects/mrx_3
collected 0 items / 1 error
================================================================================================= ERRORS ==================================================================================================
_____________________________________________________________________________ ERROR collecting tests/fuctional/test_views.py ______________________________________________________________________________
tests/fuctional/test_views.py:7: in <module>
from django.contrib.auth.models import User
../../.pyenv/versions/3.8.1/envs/MRX/lib/python3.8/site-packages/django/contrib/auth/models.py:2: in <module>
from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
../../.pyenv/versions/3.8.1/envs/MRX/lib/python3.8/site-packages/django/contrib/auth/base_user.py:48: in <module>
class AbstractBaseUser(models.Model):
../../.pyenv/versions/3.8.1/envs/MRX/lib/python3.8/site-packages/django/db/models/base.py:108: in __new__
app_config = apps.get_containing_app_config(module)
../../.pyenv/versions/3.8.1/envs/MRX/lib/python3.8/site-packages/django/apps/registry.py:253: in get_containing_app_config
self.check_apps_ready()
../../.pyenv/versions/3.8.1/envs/MRX/lib/python3.8/site-packages/django/apps/registry.py:136: in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
E django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
========================================================================================= short test summary info =========================================================================================
ERROR tests/fuctional/test_views.py - django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================================ 1 error in 0.95s =============================================================================================
It seems there is a problem with some path of the pyenv: I've tried export DJANGO_SETTINGS_MODULE=mrx_proj.settings but with no success.
The solution that I found is to configure the test runner the proper way:
The settings need to point to Django Settings: DJANGO_SETTINGS_MODULE = test_settings
pytest-django
Django Testing

How to use multiprocess in django command?

I'm tring to use ProcessPoolExecutor in django command to get some results at same time. And I tried with below codes to get it
# main codes
import json
import time
import datetime
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
from redis import Redis
from django.conf import settings
from django.core.management.base import BaseCommand
from thrift.transport import TSocket, TTransport
from thrift.protocol import TBinaryProtocol
from utils.cache import pool
from services.analysis.thrift.Analysis import Client, Dashparam
from api.analysis.models import MDashBoard
redis_con = Redis(connection_pool=pool)
class AnalysisThriftService(object):
def __init__(self):
...
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.transport.close()
class Command(BaseCommand):
help = 'python manage.py --settings "xxx"'
def add_arguments(self, parser):
parser.add_argument('--dashboard_id', type=int, help="ID")
#staticmethod
def _handle_with_thrift(dashboard_id):
try:
print(dashboard_id)
with AnalysisThriftService() as thrift_server:
dashboard_result = ...
except:
import traceback
traceback.print_exc()
def handle(self, *args, **options):
dashboard_id = options["dashboard_id"]
if dashboard_id is None:
dashboard_tables = [dashboard.id for dashboard in MDashBoard.objects.all()]
with ProcessPoolExecutor(max_workers=5) as executor:
executor.map(Command._handle_with_thrift, dashboard_tables)
else:
...
But I always get error like
Process Process-5:
Process Process-2:
Traceback (most recent call last):
File "D:\python3\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "D:\python3\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "D:\python3\lib\concurrent\futures\process.py", line 169, in _process_worker
call_item = call_queue.get(block=True)
File "D:\python3\lib\multiprocessing\queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "C:\Users\Domob\Desktop\dev\myapi\analysis\management\commands\dashboard_schedule_task.py", line 15, in <modu
le>
from api.analysis.models import MDashBoard
File "C:\Users\Domob\Desktop\dev\myapi\analysis\models.py", line 4, in <module>
from utils.models import BasicModel, StaticCharField
File "C:\Users\Domob\Desktop\dev\myapi\utils\models.py", line 9, in <module>
class BasicModel(models.Model):
File "C:\Users\Domob\Desktop\dev\venv_myapi\lib\site-packages\django\db\models\base.py", line 103, in __new__
app_config = apps.get_containing_app_config(module)
File "C:\Users\Domob\Desktop\dev\venv_myapi\lib\site-packages\django\apps\registry.py", line 252, in get_containing_ap
p_config
self.check_apps_ready()
File "C:\Users\Domob\Desktop\dev\venv_myapi\lib\site-packages\django\apps\registry.py", line 135, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
How can I get the expected results.
Great thanks.
You'll need to set up Django for the subprocesses in the subprocess initializer function.
def subprocess_setup():
django.setup()
# Could do other things here
# ...
with ProcessPoolExecutor(max_workers=5, initializer=subprocess_setup) as executor:
Got the same issue but solved it differently as suggested by this thread.
I had to explicitly pass the context to the ProcessPoolExecutor in order to get things right.
The code would look something like this
import multiprocessing
fork_context = multiprocessing.get_context('fork')
with ProcessPoolExecutor(max_workers=5, mp_context=fork_context) as executor:
...

error on execute import django in python3 shell and script

when i want do an import django i get an error
ubuntu 18.04 with a reverse proxy (nginx) and uwsgi (mode emperor actived) in virtual env with python 3.6.3 and latest django 2.2.5
test.py:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "testproject.settings")
import django
print("test")'
when i run python3 test.py i get :
(venv) :~/testproject/testproject/testproject$ python3.6 test.py
Traceback (most recent call last):
File "test.py", line 3, in <module>
from django.http import HttpResponse
File "/home/lukas/testproject/venv/lib/python3.6/site-packages/django/__init__.py", line 1, in <module>
from django.utils.version import get_version
File "/home/lukas/testproject/venv/lib/python3.6/site-packages/django/utils/version.py", line 4, in <module>
import subprocess
File "/usr/lib/python3.6/subprocess.py", line 140, in <module>
import threading
File "/usr/lib/python3.6/threading.py", line 7, in <module>
from traceback import format_exc as _format_exc
File "/usr/lib/python3.6/traceback.py", line 5, in <module>
import linecache
File "/home/lukas/testproject/venv/lib/python3.6/linecache.py", line 11, in <module>
import tokenize
File "/home/lukas/testproject/venv/lib/python3.6/tokenize.py", line 35, in <module>
from token import *
File "/home/lukas/testproject/testproject/testproject/token.py", line 1, in <module>
from django.contrib.auth.tokens import PasswordResetTokenGenerator
File "/home/lukas/testproject/venv/lib/python3.6/site-packages/django/contrib/auth/__init__.py", line 4, in <module>
from django.apps import apps as django_apps
File "/home/lukas/testproject/venv/lib/python3.6/site-packages/django/apps/__init__.py", line 2, in <module>
from .registry import apps
File "/home/lukas/testproject/venv/lib/python3.6/site-packages/django/apps/registry.py", line 426, in <module>
apps = Apps(installed_apps=None)
File "/home/lukas/testproject/venv/lib/python3.6/site-packages/django/apps/registry.py", line 46, in __init__
self.ready_event = threading.Event()
AttributeError: module 'threading' has no attribute 'Event'
i have the same error on python3 shell when i do import django whereas django have been installed with pip3 install and production mode is ok and developement mode with runserver is ok too.
can you help me, i m lost...
Do you have, by any chance, a token module in you django project?
As the python path is modified to prefer local modules rather than other pre-defined modules, the from token import * instruction will import from your module instead of the python lib one.
Try renaming your local modules to avoid collision with builtin python modules.

django celery Getting TypeError: 'tuple' object is not callable

I've just installed celery on my local machine (win10) with RabbitMQ and followed their beginners tutorial . But I get the error message TypeError: 'tuple' object is not callable
src>workon clicknstrip
(clicknstrip) src>python manage.py celery beat
celery beat v3.1.18 (Cipater) is starting.
__ - ... __ - _
Configuration ->
. broker -> amqp://guest:**#localhost:5672//
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]#%INFO
. maxinterval -> now (0s)
[2015-08-13 10:01:13,441: INFO/MainProcess] beat: Starting...
[2015-08-13 10:01:13,466: WARNING/MainProcess] DB Reset: Account for new __version__ field
[2015-08-13 10:01:13,470: CRITICAL/MainProcess] beat raised exception <type 'exceptions.TypeError'>: TypeError("'tuple' object is not callable",)
Traceback (most recent call last):
File "clicknstrip\lib\site-packages\celery\apps\beat.py", line 112, in start_scheduler
beat.start()
File "clicknstrip\lib\site-packages\celery\beat.py", line 454, in start
humanize_seconds(self.scheduler.max_interval))
File "clicknstrip\lib\site-packages\kombu\utils\__init__.py", line 322, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "clicknstrip\lib\site-packages\celery\beat.py", line 494, in scheduler
return self.get_scheduler()
File "clicknstrip\lib\site-packages\celery\beat.py", line 489, in get_scheduler
lazy=lazy)
File "clicknstrip\lib\site-packages\celery\utils\imports.py", line 53, in instantiate
return symbol_by_name(name)(*args, **kwargs)
File "clicknstrip\lib\site-packages\celery\beat.py", line 358, in __init__
Scheduler.__init__(self, *args, **kwargs)
File "clicknstrip\lib\site-packages\celery\beat.py", line 185, in __init__
self.setup_schedule()
File "clicknstrip\lib\site-packages\celery\beat.py", line 406, in setup_schedule
self.install_default_entries(self.schedule)
File "clicknstrip\lib\site-packages\celery\beat.py", line 190, in install_default_entries
not self.app.backend.supports_autoexpire:
File "clicknstrip\lib\site-packages\kombu\utils\__init__.py", line 322, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "clicknstrip\lib\site-packages\celery\app\base.py", line 625, in backend
return self._get_backend()
File "clicknstrip\lib\site-packages\celery\app\base.py", line 444, in _get_backend
return backend(app=self, url=url)
TypeError: 'tuple' object is not callable
This is my celery.py
from __future__ import absolute_import
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'clicknstrip.settings.local')
from django.conf import settings
app = Celery('clicknstrip')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
This is my task.py
from __future__ import absolute_import
from celery import shared_task
#shared_task
def add(x, y):
return x + y
#shared_task
def mul(x, y):
return x * y
#shared_task
def xsum(numbers):
return sum(numbers)
Found my issue because I added an extra comma at the end of
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',

Supervisor, Celery, Virtualenvwrapper, Django: Could not import django settings even when pythonpath added to environment

Can anyone see what I'm doing wrong here? No matter what I try, I can't seem to set it to correctly detect the settings for Django. Works fine when being run manually in terminal in the active virtualenv.
Supervisor
[program:celery_beetlejuice]
command = /home/padraic/.virtualenvs/beetlejuice/bin/python /home/padraic/CodeDev/beetlejuice/beetlejuice_django/manage.py celeryd -B -E -l INFO
directory=/home/padraic/CodeDev/beetlejuice/beetlejuice_django
environment=PYTHONPATH="/home/padraic/CodeDev/beetlejuice/beetlejuice_django", DJANGO_SETTINGS_MODULE="beetlejuice_django.settings"
user = padraic
autostart=true
autorestart=true
stdout_logfile=/home/padraic/CodeDev/beetlejuice/beetlejuice_django/logs/celeryd.log
stderr_logfile=/home/padraic/CodeDev/beetlejuice/beetlejuice_django/logs/celeryd.log
redirect_stderr=true
priority=998
numprocs=1
Traceback
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/padraic/.virtualenvs/beetlejuice/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 453, in execute_from_command_line
utility.execute()
File "/home/padraic/.virtualenvs/beetlejuice/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/padraic/.virtualenvs/beetlejuice/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 263, in fetch_command
app_name = get_commands()[subcommand]
File "/home/padraic/.virtualenvs/beetlejuice/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 109, in get_commands
apps = settings.INSTALLED_APPS
File "/home/padraic/.virtualenvs/beetlejuice/local/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
self._setup(name)
File "/home/padraic/.virtualenvs/beetlejuice/local/lib/python2.7/site-packages/django/conf/__init__.py", line 48, in _setup
self._wrapped = Settings(settings_module)
File "/home/padraic/.virtualenvs/beetlejuice/local/lib/python2.7/site-packages/django/conf/__init__.py", line 134, in __init__
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'beetlejuice_django.settings' (Is it on sys.path?):
Sigh... figured out the error...
So! I have my settings.py file which contains:
[settings.py]
import os
BEETLEJUICE_STATE = os.environ.get('BTJC_STATE', None)
if BEETLEJUICE_STATE == 'LOCAL':
from local_settings import *
elif BEETLEJUICE_STATE == 'DEV':
from development_settings import *
elif BEETLEJUICE_STATE == 'PROD':
from production_settings import *
else:
raise ImportError
<aside>
... which used to be the right way to do things (or so I was told). I'm planning to switch over to the way that Audrey and Danny recommend in Two Scoops of Django, the gist of which being:
settings/
__init__.py
base.py
local.py
test.py
production.py
... and explicitly call my settings files from now on, e.g.
python manage.py runserver --settings='settings.production'
...which imports from base.py and has all the production-specific settings.
</aside>
... and I had BTJC_STATE being exported in the virtualenvwrapper postactivate file, except that by running celeryd via supervisor I wasn't activating anything and thereby skipping that file entirely! This raised the ImportError but I didn't realise that was where it was coming from =(
Adding a string explaining the exception is a good idea... and apparently using ImproperlyConfigured is a better way to raise for this issue. So by changing the one line below in my .conf file, it started working perfectly.
[program:celery_beetlejuice]
<snip>
environment=BTJC_STATE='LOCAL'
<snip>
So that works now and I'm happy, but I need to find a good way of setting environment variables (like database passwords, api keys, and such) that will work well with for inside a non-activated in-use virtualenv.
All advice welcome on that, message me and I'll update this with the solution =)