I am trying to update a few-year old existing django project to current django and python 3. We're running into a multitude of issues, but most haven't been to hard to identify and resolve. Currently we are running into the dreaded django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. error when trying to manage.py migrate or some other commands like inspectdb.
The odd thing is that we definitely are defining the database config with a default db and engine. This setup below is from our settings.py file. It's set up that way just so that we could use sqlite for testing purposes or mysql like the live servers. I've tried both, and it doesn't work with either. I have put print statements in to confirm that it does hit the appropriate sections. I also tried the dummy engine just to test and that had the same issue.
#LOCALDEV_DB = 'MYSQL'
LOCALDEV_DB = 'SQLITE'
if LOCALDEV_DB == 'MYSQL':
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'foo',
'USER': 'root',
'PASSWORD': '123456789',
'OPTIONS': { 'init_command' : 'SET default_storage_engine=MYISAM', },
#'STORAGE_ENGINE': 'MYISAM'
}
}
elif LOCALDEV_DB == 'SQLITE':
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
The really odd thing is that when I do manage.py diffsettings it does in fact show that the database seems to be set up correctly:
DATABASES = {'default': {'ENGINE': 'django.db.backends.sqlite3', 'NAME': 'C:\\Users\\path\\to\\base\\folder\\db.sqlite3'}}
So essentially, I'm looking for any guidance on this. I've read through dozens of similar errors on SO and around the internet, and usually it's just an issue of not having a default DB or the engine defined. I seem to have both... is there something that I have forgotten or that's changed in django config in the past few years? We're jumping forward quite a few versions, so I'm hoping that I'm just missing some dumb difference in configuration in all of the documentation that I've been reading through.
Edit:
I have a tentative fix but I don't think it is legit enough to consider an actual answer. I had issues with logging initially and many solutions to the error I was getting recommended putting in a django.setup() call in the settings (though there were also people saying that this shouldn't remain in for actual deployment). That ended up fixing it that issue but eventually I ended up with this current issue. I just moved the django.setup() call beneath the database stuff, just above the logging, and now I was able to fully migrate. Any ideas why that's "fixing" stuff and what I might be able to do to avoid it?
Related
So I've got a django project with several applications.
Each application use its own database, and they share a common database containing django tables (such as auth, sessions).
For this I've got several database routers, so my settings looks like this :
DATABASES = {
'default': {
..
},
'app1_db': {
..
},
'app2_db':{
..
}
}
DATABASE_ROUTERS = ["site.db_router.App1Router", "site.db_router.App2Router"]
# no router for default database
Each application also got its unit tests. To troubleshoot my problem I extracted one of the application. It contains a tests module with four test files. Test file number one looks like below:
class ExcelTestCase(TransactionTestCase):
databases = ["app1_db"]
# some tests
Test case 1
python manage.py test app1.tests.testfile1 raises this error django.core.exceptions.ImproperlyConfigured: Circular dependency in TEST[DEPENDENCIES]
Test case 2
I comment the databases section : only default test database is created (not the application one) and this error is raised:
AssertionError: Database queries to 'app1_db' are not allowed in this test. Add 'app1_db' to app1.tests.testfile1.ExcelTestCase.databases to ensure proper test isolation and silence this fai
lure.
Test case 3
I uncomment databases setting and run python manage.py test app1.tests so every test are runned.
Three test files out of four are TransactionTestCase and therefore have databases settings.
I need to comment at least the first or third (second one is not TransactionTestCase) test file databases or I get Circulary dependency error, and by doing so I got AssertionError because I can't use the app database.
I'm using Django 2.2 and have been struggling on this issue for a while, any help will be appreciated !
Maybe setting your 'DEPENDECIES'.
controlling creation order for test databases
Check that they receive a list of dependencies and the databases without dependencies receive an EMPTY list.
Basically you have to add something like:
'default': {
..
'TEST': {
'DEPENDENCIES': ['app1_db'],
},
},
'app1_db': {
# ... db settings
'TEST': {
'DEPENDENCIES': [],
},
}
Well, I have a django project which works fine now.
I'd like to add a new app to it, in which I need to access multiple databases.
I know Django support multiple databases settings and know how to configure it. This is not the problem.
The issue is that, in the 90% of my project components, I don't need to maintain a multiple databases settings. The only usage for the second databases is in the new added app.
So I tried to alter the settings by calling:
django.conf.settings.configure(DATABASES = {....})
in the new app. And django said:
RuntimeError: Settings already configured.
Which makes sense, since I have the origin settings file and set the DJANGO_SETTINGS_MODULE up.
So I question is that what should be a good approach in this case.
I don't want to discard DJANGO_SETTINGS_MODULE variable.
I try to not include second database in the setting file initially, since the new app is also an independent module which should work independently outside the django project. So I want to have the similar config in new app to setup the database config.
Does anyone has any idea about this?
Thanks in advance!
Actually, I have the same issue in a current project. As you I have a totally independent app which uses an another database, and I could have other apps which could have the same behaviour.
The thing that I have done is to create a dir apps where I store my apps and then I add this at the end of my settings.py file :
DATABASE_ROUTERS = ['myproject.routers.MultiDBRouter']
import os
APPS_DIR = os.path.join(PROJECT_ROOT, 'apps')
for app_name in os.listdir(APPS_DIR):
print '\nLooking for settings in apps/%s :' % app_name
if os.path.exists(os.path.join(APPS_DIR, app_name, 'settings.py')):
print ' Settings file found...'
app = __import__('%s.settings' % app_name)
content = dir(app.settings)
if 'DATABASES' in content:
print ' Adding databases :'
for key, value in app.settings.DATABASES.iteritems():
if DATABASES.has_key(key):
print ' Can not add %s database config, because it already exists' % key
else:
DATABASES[key] = value
DATABASES[key]['APPS'] = [app_name]
print ' Added %s database config' % key
It will automatically look after settings.py file in all the apps/myapp/ directories. If it finds a new DATABASES variable in a app/myapp/settings.py file, it will add the other database configurations to your DATABASES variable (the true one).
I have also created a router to do not have to use the using command (the MultiDBRouter).
And then I add a settings.py file in all the app which requires another database :
DATABASES = {
'db': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'database',
'USER': 'username',
'PASSWORD': 'mysecretpassword',
}
}
So Im running into this CACHE error when I try to runserver or syncdb.
Here is the traceback: https://gist.github.com/1538051
I tried inserting this into the settings.py file:
CACHE_BACKEND = {
'default': {
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
}
}
But that gave another error which makes no sense to me.
if backend_uri.find(':') == -1:
AttributeError: 'dict' object has no attribute 'find'
Can someone help me figure what the problem is and how I can go about fixing it.
NB: I am working on the dev server
If you're using Django 1.2 or lower, CACHE_BACKEND (docs) accepts a string:
CACHE_BACKEND = 'dummy://'
If you're using Django 1.3+, CACHE_BACKEND has been deprecated in favour of CACHES (docs):
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
}
}
I'm not sure why your installation isn't creating the appropriate default - it could be that there's something else going on with your installation, but I don't really have enough information without knowing more about your settings.py etc.
First off, What version of django are you using? The dictionary- style backend config is new to django 1.3, and your traceback suggests that you are on something like 1.2.4.
If that's the case, you will need to use the older uri-style cache settings. Something like
CACHE_BACKEND = 'dummy://'
would match what you are trying to set in your question.
The full documentation on caching, relevant to Django 1.2, is available here: http://docs.djangoproject.com/en/1.2/topics/cache/
I've been evaluating django and wondered if the following is possible. I've already looked at the regular multiple database docs so please don't point me to that because this use case isn't mentioned as far as i can make out. If i'm wrong i take it back :)
I want one main database in which most of my app's models will reside, however one of the app's will need to dynamically create databases, these will be customer specific databases.
The database path (i plan to use sqlite) will be stored in primary database and so the cursor would need to be changed but the model will remain the same.
I would welcome any thoughts on ways to achieve this?
I will open with "You should not edit settings at runtime".
Having said that, I have exactly this same issue, where I want to create a unique database for each user. The reason for doing this is I am offering the ability for the user to save/access to/from a database not stored on my server, which entails having multiple databases, and thus one for each user.
This answer is NOT the recommended way to achieve the desired goal. I would love to hear from a django-guru how to best approach this problem. However, this is a solution I have been using and it has worked well so far. I am using sqlite however it can be easily modified for any of the databases.
In summary, this is the process:
Add the new database to settings (at runtime)
Create a file to store these settings for reloading when the server is restarted (at runtime)
Run a script which loads the saved settings files (whenever the server is restarted)
Now, how to achieve this:
1) Firstly, when a new user is created, I create a new database in the settings. This code lives in my view where new users are created.
from YOUR_PROJECT_NAME import settings
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
settings.DATABASES[database_id] = newDatabase
save_db_settings_to_file(newDatabase) #this is for step 2)
This script loads the database settings 'at runtime' into the django project settings. However if the server is restarted, this database will no longer be in settings.
2) To facilitate reloading these settings automatically whenever the server is restarted, I create a file for each database which will be loaded whenever the server is started. Creating this file is performed by the function save_db_settings_to_file:
def save_db_settings_to_file(db_settings):
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
newDbString = """
DATABASES['%(id)s'] = {
'ENGINE': '%(ENGINE)s', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': '%(NAME)s', # Or path to database file if using sqlite3.
'USER': '', # Not used with sqlite3.
'PASSWORD': '', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
""" % db_settings
file_to_store_settings = os.path.join(path_to_store_settings, db_settings['id'] + ".py")
write_file(file_to_store_settings, newDbString) #psuedocode for compactness
3) To actually load these settings when the server is started, I add a single line to the very bottom of /path/to/your/project/YOUR_PROJECT_NAME/settings.py, which loads each file in the settings folder and runs it, having the effect of loading the database details into the settings.
import settings_manager
Then, import settings_manager will load the file at /path/to/your/project/YOUR_PROJECT_NAME/settings_manager.py, which contains the following code:
from settings import DATABASES
import os
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
for fname in os.listdir(path_to_settings):
full_path = os.path.join(path_to_settings, fname)
f = open(full_path)
content = f.read()
f.close()
exec(content) #you'd better be sure that the file doesn't contain anything malicious
Note that you could put this code directly at the bottom of settings.py instead of the import statement, but using the import statement keeps the abstraction level of settings.py consistent.
This is a convenient way to load each database setting because to remove a database from the settings all you have to do is delete the settings file, and the next time the server restarts it won't load those details into the settings, and the database will not be accessible.
As I said, this works and I have had success using it so far, but this is NOT the ideal solution. I would really appreciate if someone could post a better solution.
What's bad about it:
It explicitly defies advice from django team not to modify settings at runtime. I do not know the reason for why this advice is given.
It uses an exec statement to load the data into settings. This should be OK, but if you get some corrupt or malicious code in one of those files you will be a sad panda.
Note that I still use the default database for auth and sessions data, but all the data from my own apps is stored in the user-specific database.
To augment #thedawnrider's answer, in some cases editing settings.DATABASES may not be enough. It might be more reliable to edit django.db.connections.databases, which serves as a cache and wrapper around settings.DATABASES.
e.g.
from django.db import connections
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
connections.databases[database_id] = newDatabase
This question is pretty outdated and should be updated.
Django support multiple databases out of the box
Here is many good tutorials [ 1, 2 ] and answers how to manage connections dynamically
Also you may simply use django-dynamic-db-router
Database connection settings may be configured on the fly by using configure method of settings object:
from django.conf import settings
dbs = settings.DATABASES.copy()
dbs['some_new_db'] = {'ENGINE': 'dummy'}
settings.configure(DATABASES=dbs)
I'd like to expose some (app-specific) settings to the admin interface, so users can change them comfortably and also not have to restart Django.
How should I go about this?
I checked out the applications on http://djangopackages.com/grids/g/live-setting/ (btw django-constance was the most appealing) but really what all these apps are doing is storing values in a database, providing a web interface to change them, and caching. Aren't the first two features already built into Django?
The biggest drawbacks I see are that none of the apps are drop-in replacements for the old location of these settings (settings.py), and require me to migrate to their notation, and often add another context processor to access them in templates.
Couldn't I just do this?
Create a model for my settings (this gives me the various types and validation)
Instantiate one such object to hold my settings (this allows the users to edit them in the admin interface) - I could dump defaults as fixtures like for other models
Wrap settings.py so it makes a database query for my settings - http://www.loose-bits.com/2011/04/extending-django-settings-with-derived.html
From my current, naive point of view the only drawbacks I see would be:
Adding or changing the available settings requires a schema migration (south). - I can live with that.
I have a model with possibly multiple instances but really only need a singleton. - That could actually be a useful feature at some point.
Performance/Caching: Looking at http://code.djangoproject.com/svn/django/trunk/django/conf/ I'd have to put a little bit of cleverness into the settings wrapper and/or model, so that model changes clear or update cached values. - doesn't seem to be rocket science.
Doing the same in another project would require a similar effort again. - I think a single dictionary constant in settings.py, holding model name(s) and field names for the lookups is all that would differ.
Wouldn't this be the best of both worlds - runtime admin (with all its perks), database backend, caching, and none of my settings.USED_TO_BE_IN_SETTINGS_DOT_PY would need any changing. Am I missing something?
AFAIK, the Django settings are supposed to be immutable. There are multiple reasons for this, the most obvious being that Django is not aware of the server's execution model (prefork / multi-threaded).
Also, you can't load the settings themselves from a Django model because the settings need to be loaded before you can use anything in the ORM.
So, basically, you have two solutions:
you can bootstrap the settings from the database by using any lower-level database access mechanism to load them; or
you can just define your settings in some other model and fetch them directly when you need them.
The first is an incredible hack and I don't suggest it. The second is much more direct and cleaner, but requires you to change your usual habits (from django.conf import settings).
The second approach is probably what's implemented by the 3rd-party apps you linked to.
From Django 1.8 docs:
You shouldn’t alter settings in your applications at runtime.
DATABASES is a dict. So you can manipulate how a dictionary:
import django.conf as conf
conf.settings.DATABASES['default']['NAME'] = 'novo_banco'
Take a look: https://bitbucket.org/bkroeze/django-livesettings
*Django-Livesettings is a project split from the Satchmo Project_. It provides the ability to configure settings via an admin interface, rather than by editing "settings.py".*
Maybe it can be helpful for you.
Honestly I get more Django when I analyze his code. In version 1.4.5 did it (following the module below):
myproject\manage.py
django\core\management__init__.py ## method - execute_manager
django\conf__init__.py ## class - LazySettings; attr - _wrapped
django\utils\functional.py ## class LazyObject; important method -
new_method_proxy
Functional option, but it has its risks. In the python "_" considers the attribute as protected.
from django.conf import settings
settings._wrapped.INSTALLED_APPS = () ## *really work*
In the following project: https://github.com/alexsilva/DJPlugins
you can see this variable being modified at runtime. the idea of the project is already working.
You can use recomended .configure() method of settings module:
from django.conf import settings
settings.configure(DEBUG=True)
settings module has additional handy features. Check docs.
You cannot directly modify the settings.py file
For example:
If u want change the database at runtime, you should Separate the configuration of the database
# Projecr_name/user_database.py
user_database = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'name',
'USER': 'admin',
'PASSWORD': '111111',
'HOST': '127.0.0.1',
'PORT': '3306'
},
'user_db': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'user1',
'USER': 'admin',
'PASSWORD': '22222',
'HOST': '127.0.0.1',
'PORT': '3306'
}
}
# Projecr_name/settings.py
from .user_database import user_database
...
DATABASES = user_database
...
Call in your logical view
# view.py
from ../Projecr_name/user_database import user_database
class Some(...):
def Some(request):
user_database['user_db']['NAME']='user2'
then you can change any setting at runtime use this way
If you have a variable in the settings.py and you want it to change at any time there is 2 ways the
First one is to make a table in database then make for it a serializer then make a view set and any time you want to get it you could send a http request to this viewset then retrieve the data
Second one is to use caching it is so fast and it is familiar to localStorage in angular