When i run my spider from terminal like this:
scrapy crawl GeneralSpider --set JOBDIR=will
everything works find and i can see the JOBDIR folder
however, when i try to set it programatically, like this:
from scrapy.utils.project import get_project_settings
myS = get_project_settings()
myS.set(myS, 'JOBDIR', "folder")
I get the following error:
Unhandled error in Deferred:
CRITICAL:twisted:Unhandled error in Deferred:
2016-02-15 17:49:35 [twisted] CRITICAL: Unhandled error in Deferred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/cmdline.py", line 150, in _run_command
cmd.run(args, opts)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 57, in run
self.crawler_process.crawl(spname, **opts.spargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/crawler.py", line 153, in crawl
d = crawler.crawl(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/crawler.py", line 70, in crawl
self.spider = self._create_spider(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/crawler.py", line 80, in _create_spider
return self.spidercls.from_crawler(self, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/spiders/__init__.py", line 50, in from_crawler
spider = cls(*args, **kwargs)
File "/bla bla bla spider.py", line 47, in __init__
myS.set(myS, 'JOBDIR', "myfolder")
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/settings/__init__.py", line 94, in set
priority = SETTINGS_PRIORITIES[priority]
exceptions.KeyError: 'myfolder'
CRITICAL:twisted:
2016-02-15 17:49:35 [twisted] CRITICAL:
The Settings API set method has this signature:
set(name, value, priority='project')
You're passing a settings object as setting name, and "myfolder" as priority, which is not expected.
Try doing this instead:
from scrapy.utils.project import get_project_settings
...
myS = get_project_settings()
myS.set('JOBDIR', "folder")
Related
I have written the following code:
spiders.test.py code:
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from wscraper.items import WscraperItem
class MySpider(BaseSpider):
name = "ExampleSpider"
allowed_domains = ["timeanddate.com"]
start_urls = ["https://www.timeanddate.com/worldclock/"]
def parse(self, response):
hxs = HtmlXPathSelector(response)
titles = hxs.select("/html/body/div[1]/div[8]/section[2]/div[1]/table/tbody").extract()
#for titles in titles:
#title = titles.select("a/text()").extract()
#link = titles.select("a/#href").extract()
print title
The code for scraper.items is:
from scrapy.item import Item, Field
class WscraperItem(Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = Field()
pass
I'm getting the following error on running the command "scrapy crawl ExampleSpider":
[boto] ERROR: Caught exception reading instance data
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/boto/utils.py", line 210, in
retry_url
r = opener.open(req, timeout=timeout)
File "/usr/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 101] Network is unreachable>
[boto] ERROR: Unable to read instance data, giving up
[scrapy] ERROR: Error downloading <GET
https://www.timeanddate.com/worldclock/>
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/scrapy/utils/defer.py", line 45,
in mustbe_deferred
result = f(*args, **kw)
File "/usr/lib/python2.7/dist-
packages/scrapy/core/downloader/handlers/__init__.py", line 41, in
download_request
return handler(request, spider)
File "/usr/lib/python2.7/dist-
packages/scrapy/core/downloader/handlers/http11.py", line 44, in
download_request
return agent.download_request(request)
d = super(CachingThreadedResolver, self).getHostByName(name, timeout)
File "/home/priyanka/.local/lib/python2.7/site-
packages/twisted/internet/base.py", line 276, in getHostByName
timeoutDelay = sum(timeout)
TypeError: 'float' object is not iterable
[scrapy] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
'downloader/exception_type_count/exceptions.TypeError': 1,
'downloader/request_bytes': 228,
'log_count/DEBUG': 2,
'log_count/ERROR': 3,
'log_count/INFO': 7,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
Spider name has to be str, not list, so:
class ExampleSpider(BaseSpider):
name = "timeandzone"
otherwise Scrapy spider loader fails to load it.
Basically, I had a compatibility issue.So, I installed Scrapy1.3.3 and this resolved the issue and yes as mentioned in the answer above spider name should be a string.
I'm getting acquainted with Scrapy and heavily relying on the interactive debugger, inspect_response().
I'm starting by scraping events off a calendar. Step 1 is to get the overview, step 2 is to get the details:
|-- Calendar List Page
| `-- Calendar Event Details Page
Here is my implementation:
class QuotesSpider(scrapy.Spider):
name = "my first spider"
def start_requests(self):
urls = ['https://www.ticketfly.com/venue/7337-output/']
for url in urls:
yield scrapy.Request(url=url, callback=self.parse_overview)
def parse_overview(self, response):
for event in response.css('.list-view-item'):
# This works fine: inspect_response(response, self)
details = event.css('span.ticket-link.primary-link a::attr(href)').extract_first()
if details is not None:
yield scrapy.Request(response.urljoin(details),
callback=self.parse_url)
def parse_url(self, response):
inspect_response(response, self) # This breaks
inspect_response() works fine on in step 1, but fails in step 2:
2017-04-22 10:30:47 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.ticketfly.com/purchase/event/1465896/tfly> (referer: https://www.ticketfly.com/venue/7337-output/)
Traceback (most recent call last):
File "//anaconda/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/Users/me/test/scrapy/test/test/spiders/my_spider.py", line 37, in parse_url
inspect_response(response, self)
File "//anaconda/lib/python2.7/site-packages/scrapy/shell.py", line 167, in inspect_response
Shell(spider.crawler).start(response=response)
File "//anaconda/lib/python2.7/site-packages/scrapy/shell.py", line 81, in start
banner=self.vars.pop('banner', ''))
File "//anaconda/lib/python2.7/site-packages/scrapy/utils/console.py", line 82, in start_python_console
shell(namespace=namespace, banner=banner)
File "//anaconda/lib/python2.7/site-packages/scrapy/utils/console.py", line 22, in wrapper
banner1=banner, user_ns=namespace, config=config)
File "//anaconda/lib/python2.7/site-packages/traitlets/config/configurable.py", line 416, in instance
'%s are being created.' % cls.__name__
MultipleInstanceError: Multiple incompatible subclass instances of InteractiveShellEmbed are being created.
2017-04-22 10:30:47 [root] DEBUG: Using default logger
2017-04-22 10:30:47 [root] DEBUG: Using default logger
2017-04-22 10:30:47 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.ticketfly.com/purchase/event/1457966/tfly> (referer: https://www.ticketfly.com/venue/7337-output/)
Traceback (most recent call last):
File "//anaconda/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/Users/me/test/scrapy/test/test/spiders/my_spider.py", line 37, in parse_url
inspect_response(response, self)
File "//anaconda/lib/python2.7/site-packages/scrapy/shell.py", line 167, in inspect_response
Shell(spider.crawler).start(response=response)
File "//anaconda/lib/python2.7/site-packages/scrapy/shell.py", line 81, in start
banner=self.vars.pop('banner', ''))
File "//anaconda/lib/python2.7/site-packages/scrapy/utils/console.py", line 82, in start_python_console
shell(namespace=namespace, banner=banner)
File "//anaconda/lib/python2.7/site-packages/scrapy/utils/console.py", line 22, in wrapper
banner1=banner, user_ns=namespace, config=config)
File "//anaconda/lib/python2.7/site-packages/traitlets/config/configurable.py", line 416, in instance
'%s are being created.' % cls.__name__
MultipleInstanceError: Multiple incompatible subclass instances of InteractiveShellEmbed are being created.
I tried peewee with flask for two days, but I failed till now. The code is as the follows:
import click
from flask import Flask
from flask.cli import FlaskGroup
from playhouse.flask_utils import FlaskDB
from models import *
from config import config
flask_db = FlaskDB()
def create_app(config_name):
application = Flask(__name__)
application.config.from_object(config[config_name])
flask_db.init_app(application)
flask_db.connect_db()
flask_db.database.create_tables([User])
flask_db.database.close()
#application.route('/')
def index():
return "hello world!"
return application
def create_cli_app(info):
return create_app("develop")
#click.group(cls=FlaskGroup, create_app=create_cli_app)
def cli():
pass
#cli.command()
def initdb():
flask_db.connect_db()
flask_db.database.create_tables([User])
flask_db.database.close()
if __name__ == "__main__":
cli()
When I run it with the CLI: python manage.py run, I got the following errors:
(venv) ➜ /Users/yw/Documents/web git:(master) ✗ p manage.py run
Traceback (most recent call last): File "manage.py", line 46, in
cli() File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/click/core.py",
line 716, in call
return self.main(*args, **kwargs) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/flask/cli.py",
line 345, in main
return AppGroup.main(self, *args, **kwargs) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/click/core.py",
line 696, in main
rv = self.invoke(ctx) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/click/core.py",
line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/click/core.py",
line 889, in invoke
return ctx.invoke(self.callback, **ctx.params) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/click/core.py",
line 534, in invoke
return callback(*args, **kwargs) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/click/decorators.py",
line 64, in new_func
return ctx.invoke(f, obj, *args[1:], **kwargs) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/click/core.py",
line 534, in invoke
return callback(*args, **kwargs) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/flask/cli.py",
line 388, in run_command
app = DispatchingApp(info.load_app, use_eager_loading=eager_loading) File
"/Users/yw/Documents/web/venv/lib/python3.4/site-packages/flask/cli.py",
line 124, in init
self._load_unlocked() File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/flask/cli.py",
line 148, in _load_unlocked
self._app = rv = self.loader() File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/flask/cli.py",
line 201, in load_app
rv = self.create_app(self) File "manage.py", line 30, in create_cli_app
return create_app("develop") File "manage.py", line 19, in create_app
flask_db.database.create_tables([User]) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/peewee.py",
line 3765, in create_tables
create_model_tables(models, fail_silently=safe) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/peewee.py",
line 5175, in create_model_tables
m.create_table(**create_table_kwargs) File "/Users/yw/Documents/web/venv/lib/python3.4/site-packages/peewee.py",
line 4845, in create_table
if db.sequences and pk is not False and pk.sequence: AttributeError: 'FlaskDB' object has no attribute 'sequences'
Indeed, I just want to initiate the DB by using flask.cli tool. As you see, if I use the command “python manage.py initdb”, I can only get the same error output as above.
So what is the meaning of "'FlaskDB' object has no attribute ‘sequences'"? What should I do?
Thanks for your help!
It seems that your User model doesn't have correct database.
With FlaskDB, your User model should inherit FlaskDB().Model
instead of defining class Meta: database = database.
database = FlaskDB()
class User(database.Model):
pass
I'm trying to add a nice admin interface to an existing Pyramid project. I created a test project using pcreate -s alchemy -s pyramid_fa fa_test and then copied all the extra files created into my project and altered them to be suitable.
Everything looks to be good and dandy until I try to add a formalchemy route:
config.formalchemy_model("/foo", package='bar',
model='bar.models.specific_models.Thingy',
**settings)
Then I get: ImportError: No module named forms
My question is: How do I fix this? Or what is the correct way to add an admin interface?
I've googled around a bunch to no avail...
Here's relevant code:
fainit.py:
from bar import models, faforms
import logging
def includeme(config):
config.include('pyramid_formalchemy')
config.include('bar.fainit')
config.include('fa.jquery')
config.include('pyramid_fanstatic')
model_view = 'fa.jquery.pyramid.ModelView'
session_factory = 'bar.models.access.DBSession'
## register session and model_view for later use
settings = {'package': 'bar',
'view': model_view,
'session_factory': session_factory,
}
config.registry.settings['bar.fa_config'] = settings
config.formalchemy_admin("/admin", models=models, forms=faforms,
**settings)
# Adding the package specific routes
config.include('shop.faroutes')
log.info('formalchemy_admin registered at /admin')
faroutes.py
from bar import models
import logging
log = logging.getLogger(__name__)
def includeme(config):
settings = config.registry.settings.get('shop.fa_settings}}', {})
config.formalchemy_model("/alerts", package='shop',
model='shop.models.super_models.Alert',
**settings)
log.info('shop.faroutes loaded')
And the traceback:
Starting subprocess with file monitor
Traceback (most recent call last):
File "../bin/pserve", line 9, in <module>
load_entry_point('pyramid==1.5a1', 'console_scripts', 'pserve')()
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid-1.5a1-py2.7.egg/pyramid/scripts/pserve.py", line 51, in main
return command.run()
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid-1.5a1-py2.7.egg/pyramid/scripts/pserve.py", line 316, in run
global_conf=vars)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid-1.5a1-py2.7.egg/pyramid/scripts/pserve.py", line 340, in loadapp
return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/PasteDeploy-1.5.0-py2.7.egg/paste/deploy/loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/PasteDeploy-1.5.0-py2.7.egg/paste/deploy/loadwsgi.py", line 272, in loadobj
return context.create()
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/PasteDeploy-1.5.0-py2.7.egg/paste/deploy/loadwsgi.py", line 710, in create
return self.object_type.invoke(self)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/PasteDeploy-1.5.0-py2.7.egg/paste/deploy/loadwsgi.py", line 146, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/PasteDeploy-1.5.0-py2.7.egg/paste/deploy/util.py", line 56, in fix_call
val = callable(*args, **kw)
File "/home/sheena/WORK/tv_guys_env/shop/shop/__init__.py", line 30, in main
includeme(config)
File "/home/sheena/WORK/tv_guys_env/shop/shop/fainit.py", line 8, in includeme
config.include('shop.fainit')
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid-1.5a1-py2.7.egg/pyramid/config/__init__.py", line 778, in include
c(configurator)
File "/home/sheena/WORK/tv_guys_env/shop/shop/fainit.py", line 24, in includeme
config.include('shop.faroutes')
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid-1.5a1-py2.7.egg/pyramid/config/__init__.py", line 778, in include
c(configurator)
File "/home/sheena/WORK/tv_guys_env/shop/shop/faroutes.py", line 12, in includeme
**settings)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid-1.5a1-py2.7.egg/pyramid/util.py", line 507, in wrapper
result = wrapped(self, *arg, **kw)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid_formalchemy-0.4.4-py2.7.egg/pyramid_formalchemy/__init__.py", line 58, in formalchemy_model
view=view, models=[model], model=model, **kwargs)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid_formalchemy-0.4.4-py2.7.egg/pyramid_formalchemy/__init__.py", line 85, in formalchemy_admin
forms = config.maybe_dotted('%s.forms' % package)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid-1.5a1-py2.7.egg/pyramid/config/__init__.py", line 848, in maybe_dotted
return self.name_resolver.maybe_resolve(dotted)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid-1.5a1-py2.7.egg/pyramid/path.py", line 316, in maybe_resolve
return self._resolve(dotted, package)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid-1.5a1-py2.7.egg/pyramid/path.py", line 323, in _resolve
return self._zope_dottedname_style(dotted, package)
File "/home/sheena/WORK/tv_guys_env/local/lib/python2.7/site-packages/pyramid-1.5a1-py2.7.egg/pyramid/path.py", line 372, in _zope_dottedname_style
__import__(used)
ImportError: No module named forms
It sounds like it's looking for you to create a forms module at shop.faroutes.forms.
I'm trying to integrate django-celery into an existing site and I'm coming up against an error that I can't seem to get fixed.
For context, I went through the Django first steps and the test project was successful, ie everything worked as it should.
Now, in my existing project, I can't get the celery worker running from the command line:
manage.py celery worker --loglevel=info --settings=myproject.settings.dev_settings
When i run that I get the following stack trace and error:
Traceback (most recent call last):
File "C:\sites\corecrm\manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 453, in execute_from_command_line
utility.execute()
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Python27\lib\site-packages\djcelery\management\commands\celery.py", line 22, in run_from_argv
['%s %s' % (argv[0], argv[1])] + argv[2:],
File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 901, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "C:\Python27\lib\site-packages\celery\bin\base.py", line 187, in execute_from_commandline
return self.handle_argv(prog_name, argv[1:])
File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 893, in handle_argv
return self.execute(command, argv)
File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 868, in execute
return cls(app=self.app).run_from_argv(self.prog_name, argv)
File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 148, in run_from_argv
return self(*args, **options)
File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 118, in __call__
ret = self.run(*args, **kwargs)
File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 220, in run
return self.target.run(*args, **kwargs)
File "C:\Python27\lib\site-packages\celery\bin\celeryd.py", line 153, in run
return self.app.Worker(**kwargs).run()
File "C:\Python27\lib\site-packages\celery\apps\worker.py", line 162, in run
self.app.loader.init_worker()
File "C:\Python27\lib\site-packages\celery\loaders\base.py", line 130, in init_worker
self.import_default_modules()
File "C:\Python27\lib\site-packages\djcelery\loaders.py", line 138, in import_default_modules
self.autodiscover()
File "C:\Python27\lib\site-packages\djcelery\loaders.py", line 141, in autodiscover
self.task_modules.update(mod.__name__ for mod in autodiscover() or ())
File "C:\Python27\lib\site-packages\djcelery\loaders.py", line 176, in autodiscover
for app in settings.INSTALLED_APPS])
File "C:\Python27\lib\site-packages\djcelery\loaders.py", line 195, in find_related_module
return importlib.import_module('%s.%s' % (app, related_name))
File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module
__import__(name)
File "C:\sites\corecrm\people\tasks.py", line 15, in <module>
from people.models import Customer, CustomerCsvFile, CustomerToTag, get_customer_from_csv_row
File "C:\sites\corecrm\people\models.py", line 163, in <module>
UserProfile._meta.get_field_by_name('username')[0]._max_length = 75
File "C:\Python27\lib\site-packages\django\db\models\options.py", line 351, in get_field_by_name
cache = self.init_name_map()
File "C:\Python27\lib\site-packages\django\db\models\options.py", line 380, in init_name_map
for f, model in self.get_all_related_m2m_objects_with_model():
File "C:\Python27\lib\site-packages\django\db\models\options.py", line 469, in get_all_related_m2m_objects_with_model
cache = self._fill_related_many_to_many_cache()
File "C:\Python27\lib\site-packages\django\db\models\options.py", line 483, in _fill_related_many_to_many_cache
for klass in get_models(only_installed=False):
File "C:\Python27\lib\site-packages\django\db\models\loading.py", line 197, in get_models
self._populate()
File "C:\Python27\lib\site-packages\django\db\models\loading.py", line 75, in _populate
self.load_app(app_name)
File "C:\Python27\lib\site-packages\django\db\models\loading.py", line 96, in load_app
models = import_module('.models', app_name)
File "C:\Python27\lib\site-packages\django\utils\importlib.py", line 35, in import_module
__import__(name)
File "C:\sites\corecrm\booking\models.py", line 17, in <module>
from people.models import Customer, UserProfile
ImportError: cannot import name Customer
To try and work out what the booking/models.py script sees in people I added the following at the start:
import people
print 'path: %s' % people.__path__
for item in dir(people):
print item
and that gives me the following output:
path: ['C:\\sites\\corecrm\\people']
__builtins__
__doc__
__file__
__name__
__package__
__path__
path: ['C:\\sites\\corecrm\\people']
__builtins__
__doc__
__file__
__name__
__package__
__path__
however, when I run manage.py shell --settings=myproject.settings.dev_settings I get the following output:
path: ['C:\\sites\\corecrm\\people']
__builtins__
__doc__
__file__
__name__
__package__
__path__
path: ['C:\\sites\\corecrm\\people']
__builtins__
__doc__
__file__
__name__
__package__
__path__
models
As you can see the models module is available at the end of the 2nd list for the shell command (and I've confirmed this is also the case for manage.py commands other than celery). How would I make sure that module is available at the same point when I run the celery command?
EDIT: I've now also set up this project on an Ubuntu VM and I'm getting the same error when I try to run the worker manage command. Any ideas? Anyone?
ANOTHER EDIT: I've pasted the code for booking/models.py and people/models.py at http://pastebin.com/fTVVBtB4
I'm pretty sure this line is your problem:
File "C:\sites\corecrm\people\models.py", line 163, in <module>
UserProfile._meta.get_field_by_name('username')[0]._max_length = 75
While you're still busy importing from people.models, this line (in particular get_field_by_name) forces Django to evaluate the model and setup all relationships between that model and it's related models. This, in turn, forces an import of Customer in people.models, while you're still busy importing that exact model. This is what results in an ImportError.
For a working solution you'll need to post your models.py.
Why does this error only occur with celery? I can't say for sure without some more information, but my best guess is that Celery handles importing everything slightly different (Django probably doesn't import Customer, CustomerCsvFile, CustomerToTag and get_customer_from_csv_row all at once) and that this exposes the bug in your code.
EDIT/SOLUTION:
I would remove this line:
UserProfile._meta.get_field_by_name('username')[0]._max_length = 75
And move it to the instance level, into the __init__ method:
class UserProfile(AbstractUser):
def __init__(self, *args, **kwargs):
self._meta.get_field_by_name('username')[0]._max_length = 75
super(UserProfile, self).__init__(*args, **kwargs)
If the cause of the issue is indeed what I think it is, this will fix the circular import while providing the same functionality. If the max_length functionality gets broken somehow (most likely because internally a max_length validator is added to CharField and _max_length is changed too late) I would instead override the complete username field in the init method:
class UserProfile(AbstractUser):
def __init__(self, *args, **kwargs):
super(UserProfile, self).__init__(*args, **kwargs)
self._meta.get_field_by_name('username')[0] = models.CharField(max_length=75, etc.)