I want to run a command for the duration of the lifetime of the website. I don't want to run it more than once.
Let's say I want to run the query:
set names utf8mb4;
Then I would just run something like:
SomeRandomObject.objects.raw('set names utf8mb4;')
Where should I put this? Does it matter what object I run the query on? Is there a better object?
I usually do this off the connection object itself.
from django.db import connections
cursor = connections['DATABASE_NAME'].cursor()
# replace DATABASE_NAME with the name of your
# DATABASE connection, i.e., the appropriate key in the
# settings.DATABASES dictionary
cursor.execute("set names utf8mb4;")
This way you can avoid having to use some random model to run your raw queries.
n.b. if you only have one database connection, i.e., default you can use:
from django.db import connection
cursor = connection.cursor()
cursor.execute("set names utf8mb4;")
To run this once at startup, you can add the following to your DATABASES
DATABASES = {
'default': {
...
'OPTIONS': {
"init_command": "set names utf8mb4;"
}
}
}
Related
I have a special case where i need to run certain testmethods against one database and other testmethods against another while maintaining the same alias (within a given TestCase).
I am not asking about multiple db connections, the issue is changing a db connection for a given alias within a testmethod, and then reverting it back to the default state after the test method executes.
I have tried the following (this code is placed within my TestCase):
from django.db import connections
from django.db.backends.postgresql.base import DatabaseWrapper
def testSwitchDBParamsForAlias(self):
db_wrapper = DatabaseWrapper(
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST' : 'host',
'PORT' : '2342',
'NAME' : 'dbname',
'USER' : 'user',
'PASSWORD': 'password'
)
connections.__setitem__('dbalias', db_wrapper)
print(connections._databases) # This still has the old connection
In the code above, 'dbAlias' already exists with different credentials and im trying to swap the credentials out using the setitem method.
Another attempt using the context manager:
def testSwitchDBParamsUsingCM(self):
with self.settings(DATABASES = DATABASES):
# call setup_databases here
print(connections._databases) #old credentials printed here
A third approach:
def testSwitchDB(self):
connections = ConnectionHandler(LAKE_TEST_DB)
print(connections._databases)
response = self.client.get(reverse('my_url'), format='json')
The third approach will actually modify the connections object! However this still does not work since in every api call (or view) the connections object is imported again and this change is overriden. So without modifying the API code, this method will also fail.
Neither works and I am left not understanding how this connection object works at all. Any ideas/suggestions?
This may be useful for you: https://docs.djangoproject.com/en/2.1/topics/db/multi-db/#manually-selecting-a-database
As long as you register both databases in your settings file, you should be able to route specific requests to either database with the using() method.
Does anyone know how to access airflow environment variable using AIRFLOW_CONN_ and use in the python code. I know we can use hook to get the password, but have been trying to use AIRFLOW_CONN in my python to connect to the database. I have saved the connection in Airflow UI and in the docs, they mentioned to use AIRFLOW_CONN_ prefix to the conn_id to use. I used it in my python code using os.environ['AIRFLOW_CONN_REDSHIFT'], but it does not identify the environment variable. Please help.
Saving the connection to database and setting an AIRFLOW_CONN_ environment variable are two different ways to add a connection. You should only choose one way, unless you want them stored under connection ids.
Assuming you are running your python code through an operator like PythonOperator, you should be able to fetch your connection just like the BaseHook does.
Stored in database:
#classmethod
def _get_connections_from_db(cls, conn_id):
session = settings.Session()
db = (
session.query(Connection)
.filter(Connection.conn_id == conn_id)
.all()
)
session.expunge_all()
session.close()
if not db:
raise AirflowException(
"The conn_id `{0}` isn't defined".format(conn_id))
return db
Stored in environment variable:
#classmethod
def _get_connection_from_env(cls, conn_id):
environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper())
conn = None
if environment_uri:
conn = Connection(conn_id=conn_id, uri=environment_uri)
return conn
Although I would recommend fetching it via a hook to avoid duplicating this code!
I am using:
A Python application in Bluemix
Bluemix cloudant v2.0.0b2 database linked to the Python app
According to https://pypi.python.org/pypi/cloudant/2.0.0b2, everything broke from 0.5 to 2.0, and they are still working on the documentation as everything is Beta. Next to this, I am also new to Python and databases. Documentation can be found here:
http://python-cloudant.readthedocs.io/en/latest/getting_started.html
What I am trying to do is check if a document already exists.
Things that I have tried:
from cloudant.account import Cloudant
import time
import json
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
doc = db['myDocument']
print doc.exists()
But the code fails before retrieving the document. I checked the source code, and it looks like it is supposed to:
def __getitem__(self, key):
if key in list(self.keys()):
return super(CouchDatabase, self).__getitem__(key)
if key.startswith('_design/'):
doc = DesignDocument(self, key)
else:
doc = Document(self, key)
if doc.exists():
doc.fetch()
super(CouchDatabase, self).__setitem__(key, doc)
return doc
else:
raise KeyError(key)
Source: https://pypi.python.org/pypi/cloudant/2.0.0b2
Is there a way I can check if the document exists before I retrieve it? Or should I retrieve it and catch the error? Or is there a different approach?
The behavior you are describing is the desired behavior for the python-cloudant library database object, so if you intend to use the database object to retrieve your documents and populate your local database cache you should look to except a KeyError in the event of a non-existent document and handle accordingly. However, if are interested in capturing whether a document exists before bringing it into your local database cache then changing your code to something like:
from cloudant.account import Cloudant
from cloudant.document import Document
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
if Document(db, 'myDocument').exists():
doc = db['myDocument']
would do the trick.
Similarly you could just do:
from cloudant.account import Cloudant
from cloudant.document import Document
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
doc = Document(db, 'myDocument')
if doc.exists():
doc.fetch()
But this would not populate your local database cache, the db dictionary.
Well, I have a django project which works fine now.
I'd like to add a new app to it, in which I need to access multiple databases.
I know Django support multiple databases settings and know how to configure it. This is not the problem.
The issue is that, in the 90% of my project components, I don't need to maintain a multiple databases settings. The only usage for the second databases is in the new added app.
So I tried to alter the settings by calling:
django.conf.settings.configure(DATABASES = {....})
in the new app. And django said:
RuntimeError: Settings already configured.
Which makes sense, since I have the origin settings file and set the DJANGO_SETTINGS_MODULE up.
So I question is that what should be a good approach in this case.
I don't want to discard DJANGO_SETTINGS_MODULE variable.
I try to not include second database in the setting file initially, since the new app is also an independent module which should work independently outside the django project. So I want to have the similar config in new app to setup the database config.
Does anyone has any idea about this?
Thanks in advance!
Actually, I have the same issue in a current project. As you I have a totally independent app which uses an another database, and I could have other apps which could have the same behaviour.
The thing that I have done is to create a dir apps where I store my apps and then I add this at the end of my settings.py file :
DATABASE_ROUTERS = ['myproject.routers.MultiDBRouter']
import os
APPS_DIR = os.path.join(PROJECT_ROOT, 'apps')
for app_name in os.listdir(APPS_DIR):
print '\nLooking for settings in apps/%s :' % app_name
if os.path.exists(os.path.join(APPS_DIR, app_name, 'settings.py')):
print ' Settings file found...'
app = __import__('%s.settings' % app_name)
content = dir(app.settings)
if 'DATABASES' in content:
print ' Adding databases :'
for key, value in app.settings.DATABASES.iteritems():
if DATABASES.has_key(key):
print ' Can not add %s database config, because it already exists' % key
else:
DATABASES[key] = value
DATABASES[key]['APPS'] = [app_name]
print ' Added %s database config' % key
It will automatically look after settings.py file in all the apps/myapp/ directories. If it finds a new DATABASES variable in a app/myapp/settings.py file, it will add the other database configurations to your DATABASES variable (the true one).
I have also created a router to do not have to use the using command (the MultiDBRouter).
And then I add a settings.py file in all the app which requires another database :
DATABASES = {
'db': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'database',
'USER': 'username',
'PASSWORD': 'mysecretpassword',
}
}
I've been evaluating django and wondered if the following is possible. I've already looked at the regular multiple database docs so please don't point me to that because this use case isn't mentioned as far as i can make out. If i'm wrong i take it back :)
I want one main database in which most of my app's models will reside, however one of the app's will need to dynamically create databases, these will be customer specific databases.
The database path (i plan to use sqlite) will be stored in primary database and so the cursor would need to be changed but the model will remain the same.
I would welcome any thoughts on ways to achieve this?
I will open with "You should not edit settings at runtime".
Having said that, I have exactly this same issue, where I want to create a unique database for each user. The reason for doing this is I am offering the ability for the user to save/access to/from a database not stored on my server, which entails having multiple databases, and thus one for each user.
This answer is NOT the recommended way to achieve the desired goal. I would love to hear from a django-guru how to best approach this problem. However, this is a solution I have been using and it has worked well so far. I am using sqlite however it can be easily modified for any of the databases.
In summary, this is the process:
Add the new database to settings (at runtime)
Create a file to store these settings for reloading when the server is restarted (at runtime)
Run a script which loads the saved settings files (whenever the server is restarted)
Now, how to achieve this:
1) Firstly, when a new user is created, I create a new database in the settings. This code lives in my view where new users are created.
from YOUR_PROJECT_NAME import settings
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
settings.DATABASES[database_id] = newDatabase
save_db_settings_to_file(newDatabase) #this is for step 2)
This script loads the database settings 'at runtime' into the django project settings. However if the server is restarted, this database will no longer be in settings.
2) To facilitate reloading these settings automatically whenever the server is restarted, I create a file for each database which will be loaded whenever the server is started. Creating this file is performed by the function save_db_settings_to_file:
def save_db_settings_to_file(db_settings):
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
newDbString = """
DATABASES['%(id)s'] = {
'ENGINE': '%(ENGINE)s', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': '%(NAME)s', # Or path to database file if using sqlite3.
'USER': '', # Not used with sqlite3.
'PASSWORD': '', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
""" % db_settings
file_to_store_settings = os.path.join(path_to_store_settings, db_settings['id'] + ".py")
write_file(file_to_store_settings, newDbString) #psuedocode for compactness
3) To actually load these settings when the server is started, I add a single line to the very bottom of /path/to/your/project/YOUR_PROJECT_NAME/settings.py, which loads each file in the settings folder and runs it, having the effect of loading the database details into the settings.
import settings_manager
Then, import settings_manager will load the file at /path/to/your/project/YOUR_PROJECT_NAME/settings_manager.py, which contains the following code:
from settings import DATABASES
import os
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
for fname in os.listdir(path_to_settings):
full_path = os.path.join(path_to_settings, fname)
f = open(full_path)
content = f.read()
f.close()
exec(content) #you'd better be sure that the file doesn't contain anything malicious
Note that you could put this code directly at the bottom of settings.py instead of the import statement, but using the import statement keeps the abstraction level of settings.py consistent.
This is a convenient way to load each database setting because to remove a database from the settings all you have to do is delete the settings file, and the next time the server restarts it won't load those details into the settings, and the database will not be accessible.
As I said, this works and I have had success using it so far, but this is NOT the ideal solution. I would really appreciate if someone could post a better solution.
What's bad about it:
It explicitly defies advice from django team not to modify settings at runtime. I do not know the reason for why this advice is given.
It uses an exec statement to load the data into settings. This should be OK, but if you get some corrupt or malicious code in one of those files you will be a sad panda.
Note that I still use the default database for auth and sessions data, but all the data from my own apps is stored in the user-specific database.
To augment #thedawnrider's answer, in some cases editing settings.DATABASES may not be enough. It might be more reliable to edit django.db.connections.databases, which serves as a cache and wrapper around settings.DATABASES.
e.g.
from django.db import connections
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
connections.databases[database_id] = newDatabase
This question is pretty outdated and should be updated.
Django support multiple databases out of the box
Here is many good tutorials [ 1, 2 ] and answers how to manage connections dynamically
Also you may simply use django-dynamic-db-router
Database connection settings may be configured on the fly by using configure method of settings object:
from django.conf import settings
dbs = settings.DATABASES.copy()
dbs['some_new_db'] = {'ENGINE': 'dummy'}
settings.configure(DATABASES=dbs)