I'm trying to follow the method here to 'Scrapy' data from the web and simultaneously save that data directly to my Django database using Scrapy's item pipeline.
However, when I try to run scrapy crawl spidername, I'm getting the error:
ImportError: No module named django.core.management
At first I thought it was because my Scrapy project was outside of my Django project folder, but even after I moved the whole project into my Django project folder I kept getting the same error. If I open a python shell inside the Scrapy project folder in its new location (inside my Django project folder), import django.core.management works fine. So what's going on?
EDIT: Some additional info: I'm doing this on a Webfaction server, and the path to my Django project is /home/gchorn/webapps/django_app/django_project. I'm using Django version 1.4.1, Scrapy 0.16 and Python2.7. The layout of the Django+Scrapy project is as follows:
django_project/
__init__.py
manage.py
settings.py
urls.py
myproject #folder containing wsgi.py
app1
app2
app3
templates
ScrapyProject/
scrapy.cfg
ScrapyProject/
__init__.py
items.py
pipelines.py
settings.py
spiders/
__init__.py
my_spider.py
Try setting this in your Spider's settings.py:
import os
import sys
sys.path.append('/home/gchorn/webapps/django_app')
os.environ['DJANGO_SETTINGS_MODULE'] = 'django_project.settings'
Then you can import your model classes like:
from django_project.app1.models import some_model
Related
I have a working Django project. If I run it - everything works fine
Here is the structure of the project
titest_project
tibrains_app
tools
__init__.py
load_django.py
countries_add.py
apps.py
models.py
load_django.py
import sys
import os
import django
sys.path.append('/mnt/HDD/tibrains/03/titest_project')
os.environ['DJANGO_SETTINGS_MODULE'] = 'titest_project.settings'
django.setup()
countries_add.py
import load_django
from tibrains_app.models import ShopCountry
When I run countries_add.py it gives the error - No module named 'decouple'. Though when I run the project itself - it works without errors.
What could be the problem?
This is my folder structure:
root:
|--Django_project
|--|-- db_app
|--|-- Django_project
|--|-- manage.py
|--Scrapy_project
|--|--Scrapy_project
|--|--|--spiders
|--|--|--settings.py
|--|--|--pipelines.py
|--|--|--items.py
|--|--|--middlewares.py
|--|--scrapy.cfg
In settings.py I have this:
import sys
import os
sys.path.append(os.path.dirname(os.path.abspath('.')))
os.environ['DJANGO_SETTINGS_MODULE'] = 'Django_project.settings'
import django
django.setup()
I've tried every possible path, including an absolute path to the project root, to the Django project, to the Django app - nothing works and I get this:
ModuleNotFoundError: No module named 'django'
Thanks for the help!
EDIT:
I should probably clarify that I'm running in a virtual environment.
In Django, I used to write populating scripts and put them in the project root directory. For example,
mysite/
mysite/
manage.py
populateA.py
The first few lines of populateA.py:
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
import django
django.setup()
...
As there are more and more populating scripts, I would like to move them to another package populate:
mysite/
mysite/
manage.py
populate/
__init__.py
populateA.py
populateB.py
populateC.py
...
However, when I run the populating scripts (python populateA.py), I got the error message: ImportError: No module named 'mysite'. How to properly set the path for DJANGO_SETTINGS_MODULE?
Since populate is already a module, run its submodules.
python -m populate.populateA
I am writing some documentation for a Django project using Sphinx.
My project (Django 1.4) looks like this:
/funproject
/documentation
# this is where sphinx files live
/source
conf.py
...
/funproject
__init__.py
settings.py
admin.py
models.py
/app1
/app2
fabfile.py
manage.py
I have Sphinx setup in the conf.py:
import sys, os
sys.path.append(os.path.join(os.path.dirname(__file__), '../..'))
os.environ['DJANGO_SETTINGS_MODULE'] = 'funproject.settings'
from funproject import settings
from django.core.management import setup_environ
setup_environ(settings)
So in my Sphinx source files I can do this:
.. automodule:: app1.models
:members:
which works fine. However, these two examples do not work:
1.
.. automodule:: funproject.models
:members:
.. automodule:: fabfile
:members:
The second works if I move the fabfile into app1 and use app1.fabfile
I'm guessing I have some issue with my conf.py but I have tried many derivations but I cannot import my fabfile.py unless it is under an app dir (though under /funproject it doesn't work either).
You might try using the fabric fabfile directory style? Perhaps it's the case that automodule doesn't look at single file modules? Docs on this other style are here
My application was working fine when I wanted to see whether I could organize my project in a better way. I read through this tutorial on structuring a django project.
Before my project structure was as follows:
camucamu
books
admin.py
models.py
views.py
__init__.py
static
templates
urls.py
views.py
settings.py
wsgi.py
__init__.py
What I wanted to do was move the books app into an apps folder. Thus I did that and changed the project structure to the following:
camucamu
apps
books
admin.py
models.py
views.py
__init__.py
static
templates
urls.py
views.py
settings.py
wsgi.py
__init__.py
I then changed the imports in views.py and admin.py
from books.models to apps.books.models.
I also changed INSTALLED_APPS in settings.py from books to apps.books.
When I then tried to run syncdb, I get the following error:
raise ImproperlyConfigured('ImportError %s: %s' % (app, e.args[0]))
django.core.exceptions.ImproperlyConfigured: ImportError apps.books: No module named apps.books
What am I messing up here so it can't find my app anymore?
Your apps folder does not have an __init__.py file so it cannot be recognized as a python module
I got the same error, following the same guide, as the last point of the following list was not cited. Make sure you performed the following changes:
Create a blank __init__.py file inside the apps folder (needed for python to recognize it as a package)
Update the import statements wherever you refer to an external app:
from projectname.apps.appname.models import YourModel, YourOtherModel
Inside settings.py edit INSTALLED_APPS such that it looks like this:
INSTALLED_APPS = (
...
# apps
'projectname.apps.appname1',
'projectname.apps.appname2',
)
This one is not specified in the guide: In all your urls.py files, update the urlpatterns!
BEFORE:
# client views
urlpatterns += patterns('appname',
...
)
AFTER:
# client views
urlpatterns += patterns('projectname.apps.appname',
...
)
Finally remember to update your changes by calling python manage.py syncdb
Hope that helped.