I am trying to connect to Postgres database using psycopg2 in a Django app but I am unable to connect. It is not throwing any exception whatsoever. I am not even able to debug it due to lack of exception throwing.
I used -
db_settings = settings.DATABASES['default']
conn = psycopg2.connect("dbname=" + db_settings['NAME'] +
" user=" db_settings['USER'] + " host=" +
db_settings['HOST'] + " password=" + db_settings['PASSWORD'])
cur = conn.cursor()
Here settings is the setting file in my Django app. Any suggestions how can I move forward and debug it so that I can find what I might be doing wrong?
to debug... open a django shell:
./manage.py shell
then in the console, import all you need and test
import psycopg2
from django.conf import settings
# have fun
or use pdb in your code
Hope this helps
Related
I am trying to use pyspark to preprocess data for the prediction model.
I get an error when I try spark.createDataFrame out of my preprocessing.Is there a way to check how processedRDD look like before making it to dataframe?
import findspark
findspark.init('/usr/local/spark')
import pyspark
from pyspark.sql import SQLContext
import os
import pandas as pd
import geohash2
sc = pyspark.SparkContext('local', 'sentinel')
spark = pyspark.SQLContext(sc)
sql = SQLContext(sc)
working_dir = os.getcwd()
df = sql.createDataFrame(data)
df = df.select(['starttime', 'latstart','lonstart', 'latfinish', 'lonfinish', 'trip_type'])
df.show(10, False)
processedRDD = df.rdd
processedRDD = processedRDD \
.map(lambda row: (row, g, b, minutes_per_bin)) \
.map(data_cleaner) \
.filter(lambda row: row != None)
print(processedRDD)
featuredDf = spark.createDataFrame(processedRDD, ['year', 'month', 'day', 'time_cat', 'time_num', 'time_cos', \
'time_sin', 'day_cat', 'day_num', 'day_cos', 'day_sin', 'weekend', \
'x_start', 'y_start', 'z_start','location_start', 'location_end', 'trip_type'])
I am getting this error:
[Stage 1:> (0 + 1) / 1]2019-10-24 15:37:56 ERROR Executor:91 - Exception in task 0.0 in stage 1.0 (TID 1)
raise AppRegistryNotReady("Apps aren't loaded yet.") django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:588)
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:571)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
at org.apache.spark.api.python.PythonRDD$$anonfun$3.apply(PythonRDD.scala:153)
at org.apache.spark.api.python.PythonRDD$$anonfun$3.apply(PythonRDD.scala:153)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
I do not understand what this have to do with importing an app
I don't know what this script has to do with Django exactly, but adding the following lines at the top of the script will probably fix this issue:
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
import django
django.setup()
Basically, you need to load your settings and populate Django’s application registry before doing anything else. You have all the information required in the Django docs.
https://docs.djangoproject.com/en/2.2/topics/settings/#calling-django-setup-is-required-for-standalone-django-usage
Instead of manualy running Hadoop I am making a python server which is using pyspack and calculate 10 times faster heavy AI algorithms on Django server. The problem I had came from SPARK-LOCAL-IP, different IP was used (the one I use to connect to a remote database vis sshtunnel). I import and use pyspark. I had to rename a file and add the correct IP.
cd /usr/local/spark/conf
touch spark-env.sh.template
mv -i spark-env.sh.template spark-env.sh
nano spark-env.sh
paste: SPARK-LOCAL_IP="127.0.1.1"
Then I had to add to my views.py sc.setLogLevel("ERROR") To see what was the real problem .Debuging of java in python can be problematic sometimes. A column was datetime instead of string and I fixed it.
Python 2.7
Django 1.10
settings.ini file(located at "/opts/myproject/settings.ini"):
[settings]
DEBUG: True
SECRET_KEY: '5a88V*GuaQgAZa8W2XgvD%dDogQU9Gcc5juq%ax64kyqmzv2rG'
On my django settings file I have:
import os
from ConfigParser import RawConfigParser
config = RawConfigParser()
config.read('/opts/myproject/settings.ini')
SECRET_KEY = config.get('settings', 'SECRET_KEY')
DEBUG = config.get('settings', 'DEBUG')
The setup works fine locally, but when I deploy to my server I get the following error if I try run any django management commands:
ConfigParser.NoSectionError: No section: 'settings'
If I go into Python shell locally I type in the above imports and read the file I get back:
['/opts/myproject/settings.ini']
On server I get back:
[]
I have tried changing "confif.read()" to "config.readfp()" as suggested on here but it didn't work.
Any help or advice is appreciated.
I created several servers, without any issue, with the stack nginx - uwsgi - flask using virtualenv.
with the current one uwsgi is throwing the error cannot import name "appl"
here is the myapp directory structure:
/srv/www/myapp
+ run.py
+ venv/ # virtualenv
+ myapp/
+ init.py
+ other modules/
+ logs/
here is the /etc/uwsgi/apps-avaliable/myapp.ini
[uwsgi]
# Variables
base = /srv/www/myapp
app = run
# Generic Config
# plugins = http, python
# plugins = python
home = %(base)/venv
pythonpath = %(base)
socket = /tmp/%n.sock
module = %(app)
callable = appl
logto = %(base)/logs/uwsgi_%n.log
and this is run.py
#!/usr/bin/env python
from myapp import appl
if __name__ == '__main__':
DEBUG = True if appl.config['DEBUG'] else False
appl.run(debug=DEBUG)
appl is defined in myapp/ _ init _ .py as an instance of Flask()
(underscores spaced just to prevent SO to turn them into bold)
I accurately checked the python code and indeed if I activate manually the virtualenvironment and execute run.py manually everything works like a charm, but uwsgi keeps throwing the import error.
Any suggestion what should I search more ?
fixed it, it was just a read permissions issue. The whole python app was readable by my user but not by the group, therefore uwsgi could not find it.
This was a bit tricky because I deployed successfully many time with the same script and never had permissions issues
So i'm getting to grips with Django, or trying to. I have some code that isn't dependent on being called by the webpage - it's designed to populate the database with information. Eventually it will be set up as a cron job to run overnight. This is the first crack at it, which is to do an initial population (once I have that working, I'll move to an add structure, where only new records are pushed.) I'm using Python 2.7, Django 1.5 and Sqlite3. When I run this code, I get
Requested setting DATABASES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
That seems fairly obvious, but I've spent a couple of hours now trying to work out how to adjust that setting. How do I call / open a connection / whatever the right terminology is here? I have a number of functions like this that will be scheduled jobs, and this has been frustrating me all afternoon.
import urllib2
import csv
import requests
from django.db import models
from gmbl.models import Match
master_data_file = urllib2.urlopen("http://www.football-data.co.uk/mmz4281/1213/E0.csv", "GET")
data = list(tuple(rec) for rec in csv.reader(master_data_file, delimiter=','))
for row in data:
current_match = Match(matchdate=row[1],
hometeam=row[2],
awayteam = row [3],
homegoals = row [4],
awaygoals = row[5],
homeshots = row[10],
awayshots = row[11],
homeshotsontarget = row[12],
awayshotsontarget = row[13],
homecorners = row[16],
awaycorners = row[17])
current_match.save()
I had originally started out with http://django-csv-importer.readthedocs.org/en/latest/ but I had the same error, and the documentation doesn't make much sense trying to debug it. When I tried calling settings.configure in the function, it said it didn't exist; presumably I had to import it, but couldn't make that work.
Make sure Django, and your project are in PYTHONPATH then you can do:
import urllib2
import csv
import requests
from django.core.management import setup_environ
from django.db import models
from yoursite import settings
setup_environ(settings)
from gmbl.models import Match
master_data_file = urllib2.urlopen("http://www.football-data.co.uk/mmz4281/1213/E0.csv", "GET")
data = list(tuple(rec) for rec in csv.reader(master_data_file, delimiter=','))
# ... your code ...
Reference: http://www.b-list.org/weblog/2007/sep/22/standalone-django-scripts/
Hope it helps!
I am trying to auto reload my django app which uses apache + mod_wsgi on my local windows machine.
I'd like to know where do I add this code that's referenced in the following article:
http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode
def _restart(path):
_queue.put(True)
prefix = 'monitor (pid=%d):' % os.getpid()
print >> sys.stderr, '%s Change detected to \'%s\'.' % (prefix, path)
print >> sys.stderr, '%s Triggering Apache restart.' % prefix
import ctypes
ctypes.windll.libhttpd.ap_signal_parent(1)
Read:
http://blog.dscpl.com.au/2008/12/using-modwsgi-when-developing-django.html
It tells you exactly where to place the file when using Django. You just need to make the code change that everyone is pointing out to you in the source code reloading documentation section related to Windows. Also read:
http://blog.dscpl.com.au/2009/02/source-code-reloading-with-modwsgi-on.html
which explains the variations on the first related to Windows.
You replace the restart function that is mentioned in the block of code above in the same article.
I use this code on my server
touch site.wsgi
and it work. After reload page in browser I get page with changes.
May be it ugly - but simple and no necessary restart apache.
In your Virtual Host config file add this:
WSGIScriptReloading On
And reload Apache
systemctl reload apache2
Enjoy!
Reference https://flask.palletsprojects.com/en/1.1.x/deploying/mod_wsgi/
You replace the restart function in the following block of code you find on the page:
Monitoring For Code Changes
The use of signals to restart a daemon process could also be employed in a mechanism which automatically detects changes to any Python modules or dependent files. This could be achieved by creating a thread at startup which periodically looks to see if file timestamps have changed and trigger a restart if they have.
Example code for such an automatic restart mechanism which is compatible with how mod_wsgi works is shown below.
import os
import sys
import time
import signal
import threading
import atexit
import Queue
_interval = 1.0
_times = {}
_files = []
_running = False
_queue = Queue.Queue()
_lock = threading.Lock()
def _restart(path):
_queue.put(True)
prefix = 'monitor (pid=%d):' % os.getpid()
print >> sys.stderr, '%s Change detected to \'%s\'.' % (prefix, path)
print >> sys.stderr, '%s Triggering process restart.' % prefix
os.kill(os.getpid(), signal.SIGINT)
I test this with Bitnami DjangoStack http://bitnami.org/stack/djangostack and Windows XP installed on D:\BitNami DjangoStack and C:\Documents and Settings\tsurahman\BitNami DjangoStack projects\myproject as project directory (default install)
as in http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Restarting_Apache_Processes, I added
MaxRequestsPerChild 1
in file D:\BitNami DjangoStack\apps\django\conf\django.conf
see comment by Graham Dumpleton
then I created a file monitor.py in my project directory with content as in http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes and replace the _restart method with http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Restarting_Windows_Apache, here is the part of the script
....
_running = False
_queue = Queue.Queue()
_lock = threading.Lock()
def _restart(path):
_queue.put(True)
prefix = 'monitor (pid=%d):' % os.getpid()
print >> sys.stderr, '%s Change detected to \'%s\'.' % (prefix, path)
print >> sys.stderr, '%s Triggering Apache restart.' % prefix
import ctypes
ctypes.windll.libhttpd.ap_signal_parent(1)
def _modified(path):
try:
....
and in file D:\BitNami DjangoStack\apps\django\scripts\django.wsgi,
....
import django.core.handlers.wsgi
import monitor
monitor.start(interval=1.0)
monitor.track(os.path.join(os.path.dirname(__file__), 'site.cf'))
application = django.core.handlers.wsgi.WSGIHandler()
and then restart the Apache server