I am trying to get a crontab task to work on a django project hosted by webfaction. I modeled it after a statement identical to one that works on a different webfaction server:
*/2 * * * * (cd /home/mb52089/webapps/smartcarpools_prod/smartcarpools; /usr/local/bin/python2.7 manage.py send_msg >> /home/mb52089/webapps/smartcarpools_prod/arch_send_msg.log 2>&1)
Getting the following error:
Traceback (most recent call last):
File "manage.py", line 14, in <module>
execute_manager(settings)
File "/home/mb52089/lib/python2.7/django/core/management/__init__.py", line 459, in execute_manager
utility.execute()
File "/home/mb52089/lib/python2.7/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/mb52089/lib/python2.7/django/core/management/__init__.py", line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/mb52089/lib/python2.7/django/core/management/__init__.py", line 70, in load_command_class
return module.Command()
AttributeError: 'module' object has no attribute 'Command'
the command also does not work if I try to run it from the command line. However, if I run python2.7 manage.py shell, that does work, so I'm getting to manage.py OK.
I've been spinning my wheels on this one for a while, and webfaction hasn't been much help. Any help is greatly appreciated. Thanks!
I made a rookie mistake and completely left out some of the necessary structure for a django command task:
class Command(BaseCommand):
help = 'Deletes expired legs, preferences and assignments'
def handle(self, *args, **options):
Related
I'm trying to import in views.py of my Django app, a python file "load_model.py" which contains my custom pyspark API but I got an error And I can't figure out how to solve it.
I import the file "load-model.py" with a simple:
import load_model as lm
My load_model.py contains the following code (this is just part of the code):
import findspark
# findspark.init('/home/student/spark-2.1.1-bin-hadoop2.7')
findspark.init('/Users/fabiomagarelli/spark-2.4.3-bin-hadoop2.7')
from pyspark.sql import SparkSession
from pyspark.ml.regression import RandomForestRegressionModel
from pyspark.ml.linalg import Vectors
from pyspark.ml.feature import VectorAssembler
from pyspark.sql import Row
from collections import OrderedDict
spark = SparkSession.builder.appName('RForest_Regression').getOrCreate()
sc = spark.sparkContext
model = RandomForestRegressionModel.load('model/')
def predict(df):
predictions = model.transform(df)
return int(predictions.select('prediction').collect()[0].prediction)
# etc... ... ...
when I lunch python manage.py run server on my command line, I get this error log:
19/07/20 07:22:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):
File "manage.py", line 21, in <module>
main()
File "manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "/anaconda3/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/anaconda3/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/anaconda3/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/anaconda3/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 60, in execute
super().execute(*args, **options)
File "/anaconda3/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/anaconda3/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 95, in handle
self.run(**options)
File "/anaconda3/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 102, in run
autoreload.run_with_reloader(self.inner_run, **options)
File "/anaconda3/lib/python3.7/site-packages/django/utils/autoreload.py", line 585, in run_with_reloader
start_django(reloader, main_func, *args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/django/utils/autoreload.py", line 570, in start_django
reloader.run(django_main_thread)
File "/anaconda3/lib/python3.7/site-packages/django/utils/autoreload.py", line 288, in run
self.run_loop()
File "/anaconda3/lib/python3.7/site-packages/django/utils/autoreload.py", line 294, in run_loop
next(ticker)
File "/anaconda3/lib/python3.7/site-packages/django/utils/autoreload.py", line 334, in tick
for filepath, mtime in self.snapshot_files():
File "/anaconda3/lib/python3.7/site-packages/django/utils/autoreload.py", line 350, in snapshot_files
for file in self.watched_files():
File "/anaconda3/lib/python3.7/site-packages/django/utils/autoreload.py", line 249, in watched_files
yield from iter_all_python_module_files()
File "/anaconda3/lib/python3.7/site-packages/django/utils/autoreload.py", line 101, in iter_all_python_module_files
modules_view = sorted(list(sys.modules.items()), key=lambda i: i[0])
RuntimeError: dictionary changed size during iteration
Exception ignored in: <function JavaWrapper.__del__ at 0x11d2de6a8>
Traceback (most recent call last):
File "/Users/fabiomagarelli/spark-2.4.3-bin-hadoop2.7/python/pyspark/ml/wrapper.py", line 41, in __del__
File "/Users/fabiomagarelli/spark-2.4.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 2000, in detach
File "/Users/fabiomagarelli/spark-2.4.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1298, in _detach
File "/Users/fabiomagarelli/spark-2.4.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 628, in _garbage_collect_object
File "/anaconda3/lib/python3.7/logging/__init__.py", line 1370, in debug
File "/anaconda3/lib/python3.7/logging/__init__.py", line 1626, in isEnabledFor
TypeError: 'NoneType' object is not callable
Exception ignored in: <function GatewayConnection.__init__.<locals>.<lambda> at 0x11da84d90>
Traceback (most recent call last):
File "/Users/fabiomagarelli/spark-2.4.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1061, in <lambda>
File "/Users/fabiomagarelli/spark-2.4.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 640, in _garbage_collect_connection
File "/Users/fabiomagarelli/spark-2.4.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 487, in quiet_shutdown
File "/anaconda3/lib/python3.7/logging/__init__.py", line 1370, in debug
File "/anaconda3/lib/python3.7/logging/__init__.py", line 1626, in isEnabledFor
TypeError: 'NoneType' object is not callable
pySpark is installed on my computer, I was using it on my jupyter notebook for fitting the model so I don't think the problem is that pyspark is not installed. Any suggestions?
So I found some tutorials on how to deploy a pySpark ML model using databricks, tensor flow etc. All too complex solutions for my limited pySpark knowledge and a project deadline in 4 weeks.
However, I found a workaround which consists in "deploying" the ML model on a Flask App then call it from my Django App (my project app). I think this may be very useful for someone facing my same problem. Not the best practice maybe but working! That's why I'm going to explain each step:
1. Create a Flask Application
in the command line (in your virtual env if you have one), type: pip install flask.
make a new folder (i call it 'static') and place in it the model folder which is obtained by saving the pySpark model (it contains other folders: data, metadata...)
create a new folder for your flask app (can be in the parent folder of your django app) and create a file in it named main.py (you can use whatever name but for the code I'm gonna post, this is the name I used).
in main.py, copy paste this:
from flask import Flask, request
import findspark
findspark.init('/home/student/spark-2.1.1-bin-hadoop2.7')
# various pySpark imports here...
app = Flask(__name__)
spark = SparkSession.builder.appName('RForest_Regression').getOrCreate()
sc = spark.sparkContext
# I'm using a RandomForest ML model, change it as appropriate
model = RandomForestRegressionModel.load('static/model/')
# define here all your functions to make a prediction (eventual arguments cleaning...)
#app.route('/predict')
# this is the function called when the page: '127.0.0.1/5000/predict' is requested.
# you can pass arguments in here by calling: '127.0.0.1/5000/predict?data=...'
numbers = request.args.get('data') # numbers = '...'
makePredictions(numbers)
def makePredictions(n):
# your function here
now on the django app, open views.py
add the fucntion to request the predictions from the flask app:
# Send a request to the flask App where the model is hosted
def getPredictions(request):
try:
data_to_predict = request.GET['data']
url = 'http://127.0.0.1:5000/predict?data=%s' % data_to_predict
response = get(url)
return JsonResponse(response.text, safe=False)
except:
print('ERROR getPredictions: no pySpark module or Flask App not running or wrong arguments')
then call the getPredictions function from javascript in your django app (I haven't done it yet so I don't have a snippet but so far is working, I tested it passing custom arguments).
You have to remember to run the flask app and the django app together in order to make it works:
cd into your Flask app folder (where you have the main.py file) then type: export FLASK_APP=main.py and flask run
then cd into your django app (where you have the manage.py file) then type: python manage.py runserver
I hope this will be useful to someone and that my explaination is not too messy. I will appreciate any comments, suggestions and requests. :)
I'm writing some unit tests for an existing django project that has had no unit tests to date.
When I run the tests from the command line using
./manage.py test
I get
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 341, in execute
django.setup()
File "/usr/local/lib/python2.7/site-packages/django/__init__.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 85, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python2.7/site-packages/django/apps/config.py", line 90, in create
module = import_module(entry)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named hijack
Now hijack is actually installed as I can use it from the browser when I run the server using
./manage.py runserver
The contents of manage.py is
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mtmdjango.settings")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
So ... how I can disable the reference for hijack when running tests or how can I enable hijack to be available under the command line when running tests?
(can someone create and add the tag django-hijack)
I worked through the tutorials at
https://docs.djangoproject.com/en/1.10/intro/tutorial01/ through https://docs.djangoproject.com/en/1.10/intro/tutorial05/
It turned out my command line was wrong.
python manage.py test contacts
gives me the output I was expecting.
I'm using Django 1.10.4 and Python 3.52. When I try to run a Django command via python manage.py my_command I get the following error:
Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "path_to_envs/envs/env_name/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "path_to_envs/envs/env_name/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "path_to_envs/envs/env_name/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File "path_to_envs/envs/env_name/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
output = self.handle(*args, **options)
TypeError: handle() got an unexpected keyword argument 'verbosity'
I can run a local django server and interact with the admin pages. The app that contains that command is in my settings.py file.
Below is the contents of the django command:
from django.core.management import BaseCommand
from my_module import MyClass
class Command(BaseCommand):
def handle(self):
my_class = MyClass()
my_class.my_method()
At the time of error, the options dictionary contains {'verbosity': 1, 'no_color': False, 'settings': None, 'pythonpath': None, 'traceback': False}. Depending on the random ordering of the dictionary no_color, traceback, and the others will throw the same TypeError. After a day of googling I still can't figure out what the issue is. Has anyone seen this before?
After lots of googling and pulling my hair out, the issue was an incorrect number of arguments to handle().
This:
def handle(self):
Should be:
def handle(self, *args, **options):
If your command needs no arguments, try a subclass of BaseCommand
NoArgsCommand.handle_noargs(**options)
I'm trying to deploy a Django (1.8) app to Heroku, runtime Python 3.4.2. The app runs succesfully (except the fact that tables are not created), but when trying run the syncdb I get the following error:
Running `python manage.py syncdb` attached to terminal... up, run.5851
/app/.heroku/python/lib/python3.4/site-packages/django/core/management
/commands/syncdb.py:24: RemovedInDjango19Warning: The syncdb command
will be removed in Django 1.9
warnings.warn("The syncdb command will be removed in Django 1.9", RemovedInDjango19Warning)
Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.4/site-packages/django/db/migrations/loader.py", line 157, in check_key
return list(self.graph.root_nodes(key[0]))[0]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/app/.heroku/python/lib/python3.4/site-packages/django /core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 390, in run_from_argv
self.execute(*args, **cmd_options)
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 441, in execute
output = self.handle(*args, **options)
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/commands/syncdb.py", line 25, in handle
call_command("migrate", **options)
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/__init__.py", line 120, in call_command
return command.execute(*args, **defaults)
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/base.py", line 441, in execute
output = self.handle(*args, **options)
File "/app/.heroku/python/lib/python3.4/site-packages/django/core/management/commands/migrate.py", line 93, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
File "/app/.heroku/python/lib/python3.4/site-packages/django/db/migrations/executor.py", line 19, in __init__
self.loader = MigrationLoader(self.connection)
File "/app/.heroku/python/lib/python3.4/site-packages/django/db/migrations/loader.py", line 47, in __init__
self.build_graph()
File "/app/.heroku/python/lib/python3.4/site-packages/django/db/migrations/loader.py", line 287, in build_graph
parent = self.check_key(parent, key[0])
File "/app/.heroku/python/lib/python3.4/site-packages/django/db/migrations/loader.py", line 164, in check_key
raise ValueError("Dependency on app with no migrations: %s" % key[0])
ValueError: Dependency on app with no migrations: authentication
I'm using sqlite.
On your dev machine, create your migrations with python manage.py makemigrations. Then, assuming your migrations folders are pushed to heroku, on heroku run python manage.py migrate instead of python manage.py syncdb. See the docs.
I hade to enter the heroku's machine bash :
heroku run bash
and then make the makemigrations:
heroku run python manage.py makemigrations
Finally,
heroku run python manage.py migrate
For me i solved this issue by using same command what we use in our local server with prefix "heroku run" that's it..
for eg:-
python manage.py makemigrations
it will work in our local but for heroku, like this :-
heroku run python manage.py makemigrations
I'm trying to setup a local django installation, but am facing a lot of problems in configuring it properly.
Trying to start the server doesn't work:
admins-macbook:myproject Admin$ python manage.py runserver 8080
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/core/management/__init__.py", line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/core/management/__init__.py", line 69, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 8, in <module>
from django.core.servers.basehttp import AdminMediaHandler, run, WSGIServerException, get_internal_wsgi_application
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 26, in <module>
from django.views import static
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/views/static.py", line 95, in <module>
template_translatable = ugettext_noop(u"Index of %(directory)s")
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/utils/translation/__init__.py", line 75, in gettext_noop
return _trans.gettext_noop(message)
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/utils/translation/__init__.py", line 48, in __getattr__
if settings.USE_I18N:
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/utils/functional.py", line 184, in inner
self._setup()
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/conf/__init__.py", line 42, in _setup
self._wrapped = Settings(settings_module)
File "/Users/Admin/myapp/lib/python2.7/site-packages/django/conf/__init__.py", line 95, in __init__
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings '{{ project_name }}.settings' (Is it on sys.path?): No module named {{ project_name }}.settings
I have mainly been following the directions here.
I am completely lost about what I may be doing wrong. I googled and found some people saying that either this could be because of naming conflicts between the project name or trying to use two installations simultaneously. I changed the name of my project, and checked to see that I only have one django installation, but still am facing the same problem. Can anyone help? Thanks for any ideas!
EDIT: Checking the paths
admins-macbook:Downloads Admin$ echo $PATH
/Users/Admin/myapp/lib/python2.7/site-packages/django/conf/myproject:/Users/Admin/myapp/lib/python2.7/site-packages/django/conf:/Users/Admin/myapp/lib/python2.7/site-packages/django:/Users/Admin/myapp/lib/python2.7/site-packages:/Users/Admin/myapp/lib/python2.7:/Users/Admin/myapp/lib:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
EDIT: Adding the directory structure (did not change, straight from django 1.4.2):
-bin
-contrib
-core
-(etc.)
-django
--conf
---__init__
---app_template
---urls
---myproject
----myapp
-----settings.py
-----urls.py
-----(etc.)
----manage.py
When manage.py is first created it includes a code that adds the path to the project's settings.py to the python path whenever a command (such as run server) is run. Since you changed the names of the folders, manage.py can't find the settings anymore.
Open up manage.py and change the old project name there to the new one. There are other places where this problem might show up so keep an eye for references to your old project's name that would cause trouble