django CLI script and database router: cannot import name connections - django

I made a database ruter for myapp application
in file /myproject/myapp/routers.py
class ShardingRouter(object):
def db_for_read(self, model, **hints):
return 'default'
def db_for_write(self, model, **hints):
return 'default'
def allow_relation(self, obj1, obj2, **hints):
return None
def allow_syncdb(self, db, model):
return None
In settings.py I have:
from django.db import connections
DATABASE_ROUTERS = ['myproject.myapp.routers.ShardingRouter',]
This works well for normal application running through wsgi, but I have one CLI script /myproject/parser_jobs.py it is starting with cron or manually from CLI:
import os, sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
...
...
if __name__ == "__main__":
do_some_long_boring_work()
now, when i run this script, I'm getting import error:
Traceback (most recent call last):
File "/hosting/myproject/myproject/parser_jobs.py", line 20, in <module>
import settings
File "/hosting/myproject/myproject/settings.py", line 46, in <module>
from django.db import connections
File "/usr/local/lib/python2.7/dist-packages/django/db/__init__.py", line 11, in <module>
if DEFAULT_DB_ALIAS not in settings.DATABASES:
File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 184, in inner
self._setup()
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 42, in _setup
self._wrapped = Settings(settings_module)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 95, in __init__
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'myproject.myproject' (Is it on sys.path?): cannot import name connections
Looks like it found settings.py, but while importing it fails on from django.db import connections. If i comment this string, it works, but without my db router:( I can add using() everywhere, but it's not cool.
So, website works good, but cli script fails. Please, help!
update: /hosting/myproject/myproject/parser_jobs.py worked good from cli, before I added DB router
pprint of sys.path in this script:
['/hosting/myproject/myproject',
'/hosting/myproject',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-linux2',
'/usr/lib/python2.7/lib-tk',
'/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/PIL',
'/usr/lib/python2.7/dist-packages/gst-0.10',
'/usr/lib/python2.7/dist-packages/gtk-2.0',
'/usr/lib/python2.7/dist-packages/ubuntu-sso-client',
'/usr/lib/pymodules/python2.7']
update: wsgi script, website works good with it:
import os
import sys
path = '/hosting/myproject'
if path not in sys.path:
sys.path.insert(0, path)
os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

my script was in the same folder with settings.py
when I moved it from /hosting/myproject/myproject/parser_jobs.py to /hosting/myproject/parser_jobs.py (where in django1.4 manage.py should be) it become working correct.
Also I changed os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings") to os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
And now it works!
In previous location parser_jobs.py works only without db router

Related

How do I test that my Celery worker actually works in Django

(code at bottom)
Context: I'm working on a Django project where I need to provide the user feedback on a task that takes 15-45 seconds. In comes Celery to the rescue! I can see that Celery is performing as expected when I celery -A my_project worker -l info & python manage.py runserver.
Problem: I can't figure out how to run a celery worker in my tests. When I run python manage.py test, I get the following error:
Traceback (most recent call last):
File "/Users/pbrockman/coding/t1v/lib/python3.8/site-packages/django/test/utils.py", line 387, in inner
return func(*args, **kwargs)
File "/Users/pbrockman/coding/tcommerce/tcommerce/tests.py", line 58, in test_shared_celery_task
self.assertEqual(result.get(), 6)
File "/Users/pbrockman/coding/t1v/lib/python3.8/site-packages/celery/result.py", line 224, in get
return self.backend.wait_for_pending(
File "/Users/pbrockman/coding/t1v/lib/python3.8/site-packages/celery/backends/base.py", line 756, in wait_for_pending
meta = self.wait_for(
File "/Users/pbrockman/coding/t1v/lib/python3.8/site-packages/celery/backends/base.py", line 1087, in _is_disabled
raise NotImplementedError(E_NO_BACKEND.strip())
NotImplementedError: No result backend is configured.
Please see the documentation for more information.
Attempted solution:
I tried various combinations of #override_settings with CELERY_TASK_ALWAYS_EAGER=True, CELERY_TASK_EAGER_PROPOGATES=True, and BROKER_BACKEND='memory'.
I tried both #app.task decorator and the #shared_task decorator.
How do I see if celery is having the expected behavior in my tests?
Code
Celery Settings: my_project/celery.py
import os
from dotenv import load_dotenv
load_dotenv()
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'my_project.settings')
app = Celery('my_project-{os.environ.get("ENVIRONMENT")}',
broker=os.environ.get('REDISCLOUD_URL'),
include=['my_project.tasks'])
from django.conf import settings
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
if __name__ == '__main__':
app.start()
Testing: my_project/tests.py
from django.test import TestCase
from tcommerce.celery import app
from tcommerce.tasks import shared_add
from tcommerce.tasks import app_add
class CeleryTests(TestCase):
def test_shared_celery_task(self):
'#shared_task'
result = shared_add.delay(2, 4)
self.assertEqual(result.get(), 6)
def test_app_celery_task(self):
'#task.app'
result = app_add.delay(2, 4)
self.assertEqual(result.get(), 6)
Defining tasks: my_project/tasks.py
from .celery import app
from celery import shared_task
#shared_task
def shared_add(x, y):
return x + y
#app.task
def app_add(x, y):
return x + y

The Wonderful Apps aren't loaded yet Error

I've searched SO and Google and can't find an answer to my problem.
I've launched my virtualenv and ran this command in the terminal:
python bin/process_messages.py
and this error occurs:
Stacktrace:
Traceback (most recent call last):
File "bin/process_messages.py", line 6, in <module>
from xyz.models import get_sku
File "/Users/myname/.environments/xyz_env/lib/python3.6/site-packages/xyz/models.py", line 19, in <module>
class Suppliers(models.Model):
File "/Users/myname/.environments/xyz_env/lib/python3.6/site-packages/django/db/models/base.py", line 110, in __new__
app_config = apps.get_containing_app_config(module)
File "/Users/myname/.environments/xyz_env/lib/python3.6/site-packages/django/apps/registry.py", line 247, in get_containing_app_config
self.check_apps_ready()
File "/Users/myname/.environments/xyz_env/lib/python3.6/site-packages/django/apps/registry.py", line 125, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
I thought it wasn't running django.setup so I added that to the script. Here is my code:
#!/usr/bin/env python
import os
import django
import boto3
from xyz.settings import SQS_QUEUE_NAME
from xyz.models import get_sku
__author__ = 'me'
def check_django_environment(default_settings):
# Environment setup for Django project files:
os.sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
if not os.environ.get('DJANGO_SETTINGS_MODULE'):
# Don't override settings if it is specified.
os.environ['DJANGO_SETTINGS_MODULE'] = default_settings
from django.conf import settings
return getattr(settings, 'DEBUG', None)
check_django_environment('xyz.settings')
django.setup()
# Get the service resource
sqs = boto3.resource('sqs')
# Get the queue
queue = sqs.get_queue_by_name(QueueName=SQS_QUEUE_NAME)
for message in queue.receive_messages():
print(message)
if message.message_attributes is not None:
print(message.message_attributes)
#sku = message.message_attributes
db_sku = get_sku(sku)
print(db_sku)
break
My Installed apps:
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'xyz',
]
By importing get_sku, you are importing your models before django.setup() has run. You need to move this import down so it happens after django.setup().
In a stand-alone script that uses Django, I generally have two groups of imports. The first contains the bare minimum to get Django setup, the second contains all the other imports, including models etc.:
#!/usr/bin/env python
import os
import django
__author__ = 'me'
def check_django_environment(default_settings):
# Environment setup for Django project files:
os.sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
if not os.environ.get('DJANGO_SETTINGS_MODULE'):
# Don't override settings if it is specified.
os.environ['DJANGO_SETTINGS_MODULE'] = default_settings
from django.conf import settings
return getattr(settings, 'DEBUG', None)
check_django_environment('xyz.settings')
django.setup()
import boto3
from xyz.settings import SQS_QUEUE_NAME
from xyz.models import get_sku
In addition to knbk's answer, you can also just use manage.py.
Here's an example script, let's say script.py:
from xyz.models import get_sku
if __name__ == '__main__':
print(get_sku)
And you run it like:
./manage.py shell < script.py
Maybe not what you're looking for, but worth knowing, nonetheless.

Django cannot find settings?

Everything was working before but all of a sudden
python manage.py runserver
throws an error
line 41, in import_module
return sys.modules[name]
KeyError: 'settings.base'
my PYTHONPATH is pointing to the root folder of my django project.
my manage.py is:
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings.base")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
I have a settings folder in which I have the base.py file. That file pulls in environment specific settings.
if os.getenv("PLATFORM") == "Heroku-Master":
from .heroku_master import *
elif os.getenv("PLATFORM") == "Heroku-Dev":
from .heroku_dev import *
elif os.getenv("PLATFORM") == "Heroku-Prod":
from .heroku_prod import *
else:
from .local import *
except Exception as e:
pass
What could be the issue ?
Is your file named settings.py?
Suppose that your project root is on /home/username/workspace/ProjectDjango and you have the following structure:
workspace
ProjectDjango
manage.py
YourAppFolder
settings.py
Your config should be:
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "YourAppFolder.settings")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)

django on cherrypy: running server.py within eclipse is OK, but not in a terminal

I followed the tutorial Tango with Django to build up my Django project. And everything runs OK.
The file structure:
tango/
rango/
tango/
wsgi.py
settings.py
manage.py
Now I am trying to deploy the project on the CherryPy server, following this tutorial. The default content of wsgi.py is as follows:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "tango.settings")
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
Now in the same folder as wsgi.py, I create the server.py:
from wsgi import application
import cherrypy
if __name__ == '__main__':
# Mount the application
cherrypy.tree.graft(application, "/")
# Unsubscribe the default server
cherrypy.server.unsubscribe()
# Instantiate a new server object
serve = cherrypy._cpserver.Server()
# Configure the server object
server.socket_host = "0.0.0.0"
server.socket_port = 8080
server.thread_pool = 30
# Subscribe this server
server.subscribe()
# Start the server engine (Option 1 *and* 2)
cherrypy.engine.start()
cherrypy.engine.block()
My question:
If I run server.py within Eclipse (Right click server.py --> Run As --> Python Run), everything works just find. However, if I enter the command $ python server.py in a terminal, the following error messages show up:
Traceback (most recent call last):
File "server.py", line 1, in <module>
from wsgi import application
File "<tangoProject>/tango/wsgi.py", line 14, in <module>
application = get_wsgi_application()
File "<virtualenv>/local/lib/python2.7/site-packages/django/core/wsgi.py", line 14, in get_wsgi_application
django.setup()
File "<virtualenv>/local/lib/python2.7/site-packages/django/__init__.py", line 20, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "<virtualenv>/local/lib/python2.7/site-packages/django/conf/__init__.py", line 46, in __getattr__
self._setup(name)
File "<virtualenv>/local/lib/python2.7/site-packages/django/conf/__init__.py", line 42, in _setup
self._wrapped = Settings(settings_module)
File "<virtualenv>/local/lib/python2.7/site-packages/django/conf/__init__.py", line 98, in __init__
% (self.SETTINGS_MODULE, e)
ImportError: Could not import settings 'tango.settings' (Is it on sys.path? Is there an import error in the settings file?): No module named tango.settings
Note, in the above I used <djangoProject> and <virtualenv> to specify the directories of the project and virtualenv, respectively.
It seemed that the server is not able to find tango/settings.py file. How do I fix it?
In your server.py before importing from wsgi add:
import sys
sys.path.append('/the/path/to/your/project')
Then, the line importing from wsgi change it to:
from tango.wsgi import application

Export django settings to my python file

I am trying to make logparser.py in django project which parses the data coming from different servers.
And on running the command on terminal :
$ python logparser.py
This error is coming :
Traceback (most recent call last):
File "logparser.py", line 13, in <module>
SMTP_CONF = settings.SMTP_CONF
File "/home/arya/.virtualenv/Devel/.virtualenvs/hu/local/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
self._setup(name)
File "/home/arya/.virtualenv/Devel/.virtualenvs/hu/local/lib/python2.7/site-packages/django/conf/__init__.py", line 48, in _setup
self._wrapped = Settings(settings_module)
File "/home/arya/.virtualenv/Devel/.virtualenvs/hu/local/lib/python2.7/site-packages/django/conf/__init__.py", line 134, in __init__
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'hma.settings' (Is it on sys.path?): No module named hma.settings
my logparser.py contains:
import re
import os
import fnmatch
import gzip
import bz2
from collections import defaultdict
from django.core.mail import send_mail
from django.core.mail.backends import smtp
import smtplib
from email.mime.text import MIMEText
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hma.settings")
from django.conf import settings
SMTP_CONF = settings.SMTP_CONF
def send_email(self,fromaddress,toaddresses,content,subject):
smtp_server = SMTP_CONF["SERVER"]
smtp_username = SMTP_CONF["USERNAME"]
smtp_password = SMTP_CONF["PASSWORD"]
smtp_port = SMTP_CONF["PORT"]
msg = MIMEText(content, 'html', _charset='utf-8')
msg['Subject'] ='Alert message for bad and internal server error'
msg['From'] = fromaddress
msg['To'] = toaddresses
server = smtplib.SMTP(smtp_server,smtp_port)
server.starttls()
server.login(smtp_username,smtp_password)
server.send_mail(fromaddress,toaddresses,msg.as_string())
server.quit()
return True
I know I am doing wrong something with command [python manage.py], but i need to run like this. Any solution for this exporting django settings to separate python file??
Well, This is the exact Usecase why Django provided an ability to create custom commands. You can use all the features of django, in your script, Its like your script will be running inside a Django Container. Here is the Documentation https://docs.djangoproject.com/en/dev/howto/custom-management-commands/.
In case you don't want to use custom management commands though there is also a simple way to run your code within Django's context. Simply put the following at the beginning of your python file, you want to run:
from django.conf import settings
from django.core import management
management.setup_environ(settings)