Django + Unix Cron, cannot import django.db - django

I am trying to have a Django script run every 5 minutes via cron on my dev laptop (Mac OS X). Here is the code in the script:
import sys
import os
def setup_environment():
pathname = os.path.dirname(sys.argv[0])
sys.path.append(os.path.abspath(pathname))
sys.path.append(os.path.normpath(os.path.join(os.path.abspath(pathname), '../')))
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
setup_environment()
from common.models import TweetCache
import datetime
def main():
print "(%s) Caching tweets..." % str(datetime.datetime.now())
previous_tweets = TweetCache.objects.filter(username='atmospherian')
for prev in previous_tweets:
prev.delete()
import twitter
api = twitter.Api()
tweets = api.GetUserTimeline('atmospherian')
for t in tweets:
tc = TweetCache(username='atmospherian', date=t.created_at, text=t.text)
tc.save()
if __name__ == '__main__':
main()
crontab:
*/5 * * * * python /absolute/path/to/tweet_cache.py
error from system mail:
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=jason>
X-Cron-Env: <USER=jason>
X-Cron-Env: <HOME=/Users/jason>
Date: Tue, 16 Feb 2010 17:45:00 -0500 (EST)
Traceback (most recent call last):
File "/Users/jason/Development/bandistry/tweet_cache.py", line 22, in <module>
from common.models import TweetCache
File "/Users/jason/Development/bandistry/common/models.py", line 1, in <module>
from django.db import models
ImportError: No module named django.db
can anyone tell me what im doing wrong?

sys.argv[0] isn't always the most reliable way to get the current file's path. I recommend this modification:
pathname = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, pathname)
sys.path.insert(0, os.path.abspath(os.path.join(pathname, '..')))
Note the use of sys.path.insert instead of sys.path.append, and the use of file. Also, using abspath on file -before- dirname reduces the chance you reduce the entire filename down to the empty string or simply '.' which may not even be accurate.
Also, is the django package installed in one of those two paths you add? If not, you sill need to add that path
Finally, a minor quip, probably unrelated to your django import issue, but you really should be doing:
os.environ['DJANGO_SETTINGS_MODULE'] = 'bandistry.settings'
It may work without, but it's better if you put your entire django application in a package. This reduces the chance of you obscuring other package names.

Related

Getting an error in django-admin.py

I have installed django 1.6 using pip on Windows 7. When I try to run django-admin.py, I get this error
C:\django-admin.py startproject test007
Traceback (most recent call last):
File "C:\Python27\Scripts\django-admin.py", line 2 in ?
from django.core import management
File "C:\Python27\Lib\site-packages\django\core\management\__init__.py", line 55
except ImportError as e:
^
SyntaxError: invalid syntax
in the \management__init__.py, the imports are
import collections
import os
import sys
from optparse import OptionParser, NO_DEFAULT
import imp
from django.core.exceptions import ImproperlyConfigured
from django.core.management.base import BaseCommand, CommandError, handle_default_options
from django.core.management.color import color_style
from django.utils.importlib import import_module
from django.utils import six
# For backwards compatibility: get_version() used to be in this module.
from django import get_version
Here's the relevant block from the same file
parts = app_name.split('.')
parts.append('management')
parts.reverse()
part = parts.pop()
path = None
# When using manage.py, the project module is added to the path,
# loaded, then removed from the path. This means that
# testproject.testapp.models can be loaded in future, even if
# testproject isn't in the path. When looking for the management
# module, we need look for the case where the project name is part
# of the app_name but the project directory itself isn't on the path.
try:
f, path, descr = imp.find_module(part, path)
except ImportError as e:
if os.path.basename(os.getcwd()) != part:
raise e
else:
if f:
f.close()
while parts:
part = parts.pop()
f, path, descr = imp.find_module(part, [path] if path else None)
if f:
f.close()
return path
in which the same line 55 from the traceback is in the try/except block towards the bottom. I've uninstalled and reinstalled but, to no avail.
It works when I give the/full/path/to/django-admin.py but it shouldn't be required.

Export django settings to my python file

I am trying to make logparser.py in django project which parses the data coming from different servers.
And on running the command on terminal :
$ python logparser.py
This error is coming :
Traceback (most recent call last):
File "logparser.py", line 13, in <module>
SMTP_CONF = settings.SMTP_CONF
File "/home/arya/.virtualenv/Devel/.virtualenvs/hu/local/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
self._setup(name)
File "/home/arya/.virtualenv/Devel/.virtualenvs/hu/local/lib/python2.7/site-packages/django/conf/__init__.py", line 48, in _setup
self._wrapped = Settings(settings_module)
File "/home/arya/.virtualenv/Devel/.virtualenvs/hu/local/lib/python2.7/site-packages/django/conf/__init__.py", line 134, in __init__
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'hma.settings' (Is it on sys.path?): No module named hma.settings
my logparser.py contains:
import re
import os
import fnmatch
import gzip
import bz2
from collections import defaultdict
from django.core.mail import send_mail
from django.core.mail.backends import smtp
import smtplib
from email.mime.text import MIMEText
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hma.settings")
from django.conf import settings
SMTP_CONF = settings.SMTP_CONF
def send_email(self,fromaddress,toaddresses,content,subject):
smtp_server = SMTP_CONF["SERVER"]
smtp_username = SMTP_CONF["USERNAME"]
smtp_password = SMTP_CONF["PASSWORD"]
smtp_port = SMTP_CONF["PORT"]
msg = MIMEText(content, 'html', _charset='utf-8')
msg['Subject'] ='Alert message for bad and internal server error'
msg['From'] = fromaddress
msg['To'] = toaddresses
server = smtplib.SMTP(smtp_server,smtp_port)
server.starttls()
server.login(smtp_username,smtp_password)
server.send_mail(fromaddress,toaddresses,msg.as_string())
server.quit()
return True
I know I am doing wrong something with command [python manage.py], but i need to run like this. Any solution for this exporting django settings to separate python file??
Well, This is the exact Usecase why Django provided an ability to create custom commands. You can use all the features of django, in your script, Its like your script will be running inside a Django Container. Here is the Documentation https://docs.djangoproject.com/en/dev/howto/custom-management-commands/.
In case you don't want to use custom management commands though there is also a simple way to run your code within Django's context. Simply put the following at the beginning of your python file, you want to run:
from django.conf import settings
from django.core import management
management.setup_environ(settings)

Deploying Django project using mod_wsgi and virtualenv

Trying to deploy a Django 1.4 project using mod_wsgi and virtualenv, i'm running into a 500. The Apache error_log reveals:
mod_wsgi (pid=30452): Exception occurred processing WSGI script '/path/to/project/site-packages/projectapp/wsgi.py'.
[...] Traceback (most recent call last):
[...] File "/path/to/project/env/myenv/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 219, in __call__
[...] self.load_middleware()
[...] File "/path/to/project/env/myenv/lib/python2.6/site-packages/django/core/handlers/base.py", line 47, in load_middleware
[...] raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e))
[...] ImproperlyConfigured: Error importing middleware projectapp.content.middleware: "cannot import name SomeModel"
From the error message i would expect that this is some kind of a path issue. However, when the offending middleware is removed from the Django settings, the site launches just fine, and there are other modules loaded from projectapp.content, SomeModel is also loaded in this case, otherwise the whole site wouldn't be able to run.
The import error raised doesn't come directly from the middleware, as it doesn't import the model. SomeModel is defined in a speparate app which is actually checked out into the src directory of the virtualenv. But the directory containing this app is also in the path.
The wsgi.py file i'm using:
import os
import sys
sys.stdout = sys.stderr
sys.path.insert(0, '/path/to/project/env/myenv/lib/python2.6/site-packages/')
# The module inside the following directory
# defines SomeModel from the error message
sys.path.insert(0, '/path/to/project/env/myenv/src/some-app/')
sys.path.insert(0, '/path/to/project/site-packages/')
import django.core.handlers.wsgi
os.environ['DJANGO_SETTINGS_MODULE'] = 'projectapp.settings'
application = django.core.handlers.wsgi.WSGIHandler()
Printing sys.path after inserting the module paths shows everything in the expected order, /path/to/project/site-packages/ is listed first and /path/to/project/env/myenv/src/some-app/ (which defines SomeModel) second.
I've also tried a different variant, based on the example from the mod_wsgi docs:
import os
import sys
import site
ALLDIRS = [
'/path/to/project/site-packages/',
'/path/to/project/env/myenv/lib/python2.6/site-packages/',
'/path/to/project/env/myenv/src/some-app/',
]
# Remember original sys.path
prev_sys_path = list(sys.path)
sys.stdout = sys.stderr
# Add each new site-packages directory
for directory in ALLDIRS:
site.addsitedir(directory)
# Reorder sys.path so new directories are at the front
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
#activate_this = '/path/to/project/env/myenv/bin/activate_this.py'
#execfile(activate_this, dict(__file__=activate_this))
import django.core.handlers.wsgi
os.environ['DJANGO_SETTINGS_MODULE'] = 'projectapp.settings'
application = django.core.handlers.wsgi.WSGIHandler()
The error logged by Apache is exactly the same.
I have to add that the middleware is loaded just fine when i'm running the development server on the same machine, so i have no idea what's going wrong. Is there a way to get a better traceback from mod_wsgi?
EDIT:
Fogot to mention that i'm using mod_wsgi in daemon mode. Here are the relevant parts from my vhost config:
<VirtualHost x.x.x.x:80>
WSGIDaemonProcess foo user=foo threads=10 umask=0002
WSGIProcessGroup foo
WSGIScriptAlias / /path/to/project/site-packages/projectapp/wsgi.py
</VirtualHost>
Okay after hours of debugging this turned out to be a race condition caused by the middleware. The middleware i'm using is similiar to the FlatpageFallbackMiddleware from Django contrib and actually the import of the view caused the problem.
from projectapp.content.views import generic_content_detail
class ContentFallbackMiddleware(object):
def process_response(self, request, response):
[...]
Moving the import statement inside the process_response method solved the problem for me:
class ContentFallbackMiddleware(object):
def process_response(self, request, response):
from projectapp.content.views import generic_content_detail
[...]

Pandas + Django + mod_wsgi + virtualenv

Pandas is producing 'module' object has no attribute 'core' when being imported under django and mod_wsgi inside a virtual environment. It works fine running under the django development server inside the virtual environment.
Other modules e.g.: numpy have no problems so I assume this means the virtual environment is set up correctly with mod_wsgi. Any advice would be appreciated.
staging.wsgi
import os
import sys
import site
PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
site_packages = os.path.join(PROJECT_ROOT, 'env/openportfolio/lib/python2.7/site-packages')
site.addsitedir(os.path.abspath(site_packages))
sys.path.insert(0, PROJECT_ROOT)
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
os.environ['DJANGO_SETTINGS_MODULE'] = 'openportfolio.settings_staging'
import pandas #triggers error
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
Error
Traceback (most recent call last):
File "/usr/local/web/django/www/staging/openportfolio/apache/staging.wsgi", line 22, in <module>
import pandas
File "/usr/local/web/django/www/staging/env/openportfolio/lib/python2.7/site-packages/pandas/__init__.py", line 12, in <module>
from pandas.core.api import *
File "/usr/local/web/django/www/staging/env/openportfolio/lib/python2.7/site-packages/pandas/core/api.py", line 6, in <module>
import pandas.core.datetools as datetools
AttributeError: 'module' object has no attribute 'core'
Python Path
['/usr/local/web/django/www/staging/openportfolio',
'/usr/local/web/django/www/staging',
'/Library/Python/2.7/site-packages/pip-1.0.2-py2.7.egg',
'/usr/local/web/django/www/staging/env/openportfolio/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg',
'/usr/local/web/django/www/staging/env/openportfolio/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg',
'/usr/local/web/django/www/staging/env/openportfolio/lib/python2.7/site-packages/matplotlib-1.1.0-py2.7-macosx-10.7-intel.egg',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
'/Library/Python/2.7/site-packages',
'/usr/local/web/django/www/staging/env/openportfolio/lib/python2.7/site-packages']
So it turns out this was a Python path order issue. By running sys.path.reverse() in my wsgi config file, the code now runs.
Due to the order of Python path, the built in OS X numpy library must have been imported first over the virtual environment one causing the issue.
'RuntimeError: module compiled against API version 6 but this version of numpy is 4' was the error line I missed in my original post which could have helped debug the answer.

Pydev and Django: Shell not finding certain modules?

I am developing a Django project with PyDev in Eclipse. For a while, PyDev's Django Shell worked great. Now, it doesn't:
>>> import sys; print('%s %s' % (sys.executable or sys.platform, sys.version))
C:\Python26\python.exe 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)]
>>>
>>> from django.core import management;import mysite.settings as settings;management.setup_environ(settings)
Traceback (most recent call last):
File "<console>", line 1, in <module>
ImportError: No module named mysite.settings
>>>
The dev server runs just fine. What could I be doing wrong?
The models module is also conspicuously absent:
>>> import mysite.myapp.models
Traceback (most recent call last):
File "<console>", line 1, in <module>
ImportError: No module named mysite.myapp.models
On the normal command line, outside of PyDev, the shell works fine.
Why could this be happening?
Seems like a simple path issue. What's the output of this:
import sys; print sys.path
I don't know anything about PyDev, but there's probably a setting somewhere to add paths to the PYTHONPATH setting. If not, you can do it directly in the shell:
sys.path.insert(0, '/path/to/directory/containing/mysite/')
I had a similar problem to this a while ago while moving my project from Django 1.3 and having the settings.py file at the root of my source and then moving it down into the application.
For example what happened was that I had the following:
rootOfSource/
- settings.py
- myapp
and I changed it to be:
rootOfSource/
- myapp
- myapp/settings.py
and I also changed my settings file to be the following:
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
However when I debugged into the os.eviron I found that the DJANGO_SETTINGS_MODULE was not as expected, I then changed my manage.py to be the following:
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")
os.environ.__setitem__("DJANGO_SETTINGS_MODULE", "myapp.settings")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
Which then allowed me to run from PyDev.
Hope this helps.
I fixed this problem by going to the project properties -> PyDev Django and setting the Django settings module.