Django .save(using='...') correct database name - django

My Django settings.py contains the following:
DATABASES = {
'default': { # several settings omitted
'NAME': 'myproject', 'HOST': 'localhost', },
'other': { 'NAME': 'other', 'HOST': 'else.where', }
}
I now want to fetch some objects from the other DB and save it to default like so:
things = Thing.objects.using('other').raw('SELECT code,created FROM other.otherapp_bigthing WHERE created>...')
# 'code' is a unique string in other and PK of the Thing model
for t in things:
t.save(using='default')
This gives me
ProgrammingError: (1146, "Table 'other.myapp_thing' doesn't exist")
which is a correct observation, however, by the documentation of the using parameter I expected the records to be saved to myproject.myapp_thing. Why is the database name still taken from the other configuration when I explicitly advised it to use default?

I eventually worked around this by making a raw insert (connections['default'].cursor().execute('INSERT INTO myproject.myapp_thing (...) ON DUPLICATE KEY UPDATE...') but still wanted to know why the question's code does not work.
The problem was that the Thing model contained another field: next to the code CharField/PK and created date (in my use case actually last_accessed) there was another date (think modified) with default=None, null=True and I assumed the default would be used if I do not explicitly include in the original raw query.
Apparently, this assumption was wrong and Django tried, on t.save, to look up a value for modified in the DB it originally came from -- and failed, since it came from a raw SQL query.
If I include ,NULL AS 'modified' in the raw SQL query the original error goes away but I still cannot reasonably use Django logic for my use case since t.save(using='default') would, for existing things, update last_accessed and overwrite the modified date with NULL, which I do not want, and t.save(using='default', update_fields=['last_accessed']) would force an update and thus fail for new things.

Related

How can i modify a db connection for a specific testmethod in a Django TestCase (and then restore upon completion)?

I have a special case where i need to run certain testmethods against one database and other testmethods against another while maintaining the same alias (within a given TestCase).
I am not asking about multiple db connections, the issue is changing a db connection for a given alias within a testmethod, and then reverting it back to the default state after the test method executes.
I have tried the following (this code is placed within my TestCase):
from django.db import connections
from django.db.backends.postgresql.base import DatabaseWrapper
def testSwitchDBParamsForAlias(self):
db_wrapper = DatabaseWrapper(
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST' : 'host',
'PORT' : '2342',
'NAME' : 'dbname',
'USER' : 'user',
'PASSWORD': 'password'
)
connections.__setitem__('dbalias', db_wrapper)
print(connections._databases) # This still has the old connection
In the code above, 'dbAlias' already exists with different credentials and im trying to swap the credentials out using the setitem method.
Another attempt using the context manager:
def testSwitchDBParamsUsingCM(self):
with self.settings(DATABASES = DATABASES):
# call setup_databases here
print(connections._databases) #old credentials printed here
A third approach:
def testSwitchDB(self):
connections = ConnectionHandler(LAKE_TEST_DB)
print(connections._databases)
response = self.client.get(reverse('my_url'), format='json')
The third approach will actually modify the connections object! However this still does not work since in every api call (or view) the connections object is imported again and this change is overriden. So without modifying the API code, this method will also fail.
Neither works and I am left not understanding how this connection object works at all. Any ideas/suggestions?
This may be useful for you: https://docs.djangoproject.com/en/2.1/topics/db/multi-db/#manually-selecting-a-database
As long as you register both databases in your settings file, you should be able to route specific requests to either database with the using() method.

Mongoengine create_user function generating null objectId's

It always seems to get the exact same error after I've added a single user to the DB.
Tried to save duplicate unique keys (E11000 duplicate key error index: mydb.user.$objectId_1 dup key: { : null })
What's strange is that I copied the code EXACTLY from the test suite and even that doesn't work. I look at the stack trace and the offending issue is always in the same place
/usr/local/lib/python2.7/dist-packages/mongoengine/django/auth.py in set_password
264. self.save()
It's incredibly frustrating and I have been looking at this for basically the last 2 days. It's not something in my code. For some reason it doesn't seem to be generating an ObjectId as it's always null. I don't even understand why that's the case
My code is simply
from mongoengine.django.auth import User
from django.contrib.auth import get_user_model
user_data = {
'username': 'user',
'email': 'user#example.com',
'password': 'test',
}
manager = get_user_model()._default_manager
user = manager.create_user(user_data)
Looks like you have an extra index on an objectId field that is Unique
When creating a new User its not setting an objectId field value and as its unique you can't have two documents without a value (as its not a sparse index).
To test in the mongo shell:
> use mydb
> db.user.getIndexes()
Drop the objectId_1 index:
> db.user.dropIndex("objectId_1")
> db.user.getIndexes()

How to fix Django error: " 'unicode' object has no attribute 'tzinfo' " on database read

I am deploying my Django project on Windows Azure. I was using SQLite as a database server and everything was ok. When I have deployed My project, I decided to connect it with an SQL Azure Database but it seems that this solution created some errors. I am no longer able to edit users profiles. I get always this error :
AttributeError at /admin/auth/user/1/
'unicode' object has no attribute 'tzinfo'
This error happens when your database contains date-time stamps like this:
0000-00-00 00:00:00.000000
(this can happen in MySQL if you delete or overwrite a previous date with MySQLWorkbench)
When you try to retrieve these records in a Django model object, you will get an exception from the pytz timezone library:
AttributeError 'unicode' object has no attribute 'tzinfo'
You should edit these dates in your database first, and set them to more recent dates, like 2018-01-01 00:00:00.000000 or set to NULL (but not blank).
See:
https://groups.google.com/forum/#!topic/django-users/Jg_9fQ3jMcU
See also:
#1292 - Incorrect date value: '0000-00-00'
Error in mysql when setting default value for DATE or DATETIME
How to store NULL values in datetime fields in MySQL?
I had same issue trying to use django-pyodbc-azure database backend with Django (1.5.1): by default it stores DateTimeField fields in your DB as datetime2(7), which looks to be still unsupported in Django.
In my case I added the option 'use_legacy_datetime' : True in settings.py, like below:
DATABASES = {
'default': {
'ENGINE' : 'sql_server.pyodbc',
'NAME' : '<MYDBNAME>',
'USER': '<MYDBUSER>',
'PASSWORD': '<MYDBPWD>',
'HOST': '<MYHOST>',
'OPTIONS': {
'use_mars': True,
'use_legacy_datetime' : True, # I added this line
},
},
}
I found my solution here.
I don't know Azure platform well, so I don't know if this is exactly your case, if not you can still modify your database replacing datetime2(N) fields with good old datetime ones.
Hope it helps.

Django multiple and dynamic databases

I've been evaluating django and wondered if the following is possible. I've already looked at the regular multiple database docs so please don't point me to that because this use case isn't mentioned as far as i can make out. If i'm wrong i take it back :)
I want one main database in which most of my app's models will reside, however one of the app's will need to dynamically create databases, these will be customer specific databases.
The database path (i plan to use sqlite) will be stored in primary database and so the cursor would need to be changed but the model will remain the same.
I would welcome any thoughts on ways to achieve this?
I will open with "You should not edit settings at runtime".
Having said that, I have exactly this same issue, where I want to create a unique database for each user. The reason for doing this is I am offering the ability for the user to save/access to/from a database not stored on my server, which entails having multiple databases, and thus one for each user.
This answer is NOT the recommended way to achieve the desired goal. I would love to hear from a django-guru how to best approach this problem. However, this is a solution I have been using and it has worked well so far. I am using sqlite however it can be easily modified for any of the databases.
In summary, this is the process:
Add the new database to settings (at runtime)
Create a file to store these settings for reloading when the server is restarted (at runtime)
Run a script which loads the saved settings files (whenever the server is restarted)
Now, how to achieve this:
1) Firstly, when a new user is created, I create a new database in the settings. This code lives in my view where new users are created.
from YOUR_PROJECT_NAME import settings
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
settings.DATABASES[database_id] = newDatabase
save_db_settings_to_file(newDatabase) #this is for step 2)
This script loads the database settings 'at runtime' into the django project settings. However if the server is restarted, this database will no longer be in settings.
2) To facilitate reloading these settings automatically whenever the server is restarted, I create a file for each database which will be loaded whenever the server is started. Creating this file is performed by the function save_db_settings_to_file:
def save_db_settings_to_file(db_settings):
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
newDbString = """
DATABASES['%(id)s'] = {
'ENGINE': '%(ENGINE)s', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': '%(NAME)s', # Or path to database file if using sqlite3.
'USER': '', # Not used with sqlite3.
'PASSWORD': '', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
""" % db_settings
file_to_store_settings = os.path.join(path_to_store_settings, db_settings['id'] + ".py")
write_file(file_to_store_settings, newDbString) #psuedocode for compactness
3) To actually load these settings when the server is started, I add a single line to the very bottom of /path/to/your/project/YOUR_PROJECT_NAME/settings.py, which loads each file in the settings folder and runs it, having the effect of loading the database details into the settings.
import settings_manager
Then, import settings_manager will load the file at /path/to/your/project/YOUR_PROJECT_NAME/settings_manager.py, which contains the following code:
from settings import DATABASES
import os
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
for fname in os.listdir(path_to_settings):
full_path = os.path.join(path_to_settings, fname)
f = open(full_path)
content = f.read()
f.close()
exec(content) #you'd better be sure that the file doesn't contain anything malicious
Note that you could put this code directly at the bottom of settings.py instead of the import statement, but using the import statement keeps the abstraction level of settings.py consistent.
This is a convenient way to load each database setting because to remove a database from the settings all you have to do is delete the settings file, and the next time the server restarts it won't load those details into the settings, and the database will not be accessible.
As I said, this works and I have had success using it so far, but this is NOT the ideal solution. I would really appreciate if someone could post a better solution.
What's bad about it:
It explicitly defies advice from django team not to modify settings at runtime. I do not know the reason for why this advice is given.
It uses an exec statement to load the data into settings. This should be OK, but if you get some corrupt or malicious code in one of those files you will be a sad panda.
Note that I still use the default database for auth and sessions data, but all the data from my own apps is stored in the user-specific database.
To augment #thedawnrider's answer, in some cases editing settings.DATABASES may not be enough. It might be more reliable to edit django.db.connections.databases, which serves as a cache and wrapper around settings.DATABASES.
e.g.
from django.db import connections
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
connections.databases[database_id] = newDatabase
This question is pretty outdated and should be updated.
Django support multiple databases out of the box
Here is many good tutorials [ 1, 2 ] and answers how to manage connections dynamically
Also you may simply use django-dynamic-db-router
Database connection settings may be configured on the fly by using configure method of settings object:
from django.conf import settings
dbs = settings.DATABASES.copy()
dbs['some_new_db'] = {'ENGINE': 'dummy'}
settings.configure(DATABASES=dbs)

Changing Django settings at runtime

I'd like to expose some (app-specific) settings to the admin interface, so users can change them comfortably and also not have to restart Django.
How should I go about this?
I checked out the applications on http://djangopackages.com/grids/g/live-setting/ (btw django-constance was the most appealing) but really what all these apps are doing is storing values in a database, providing a web interface to change them, and caching. Aren't the first two features already built into Django?
The biggest drawbacks I see are that none of the apps are drop-in replacements for the old location of these settings (settings.py), and require me to migrate to their notation, and often add another context processor to access them in templates.
Couldn't I just do this?
Create a model for my settings (this gives me the various types and validation)
Instantiate one such object to hold my settings (this allows the users to edit them in the admin interface) - I could dump defaults as fixtures like for other models
Wrap settings.py so it makes a database query for my settings - http://www.loose-bits.com/2011/04/extending-django-settings-with-derived.html
From my current, naive point of view the only drawbacks I see would be:
Adding or changing the available settings requires a schema migration (south). - I can live with that.
I have a model with possibly multiple instances but really only need a singleton. - That could actually be a useful feature at some point.
Performance/Caching: Looking at http://code.djangoproject.com/svn/django/trunk/django/conf/ I'd have to put a little bit of cleverness into the settings wrapper and/or model, so that model changes clear or update cached values. - doesn't seem to be rocket science.
Doing the same in another project would require a similar effort again. - I think a single dictionary constant in settings.py, holding model name(s) and field names for the lookups is all that would differ.
Wouldn't this be the best of both worlds - runtime admin (with all its perks), database backend, caching, and none of my settings.USED_TO_BE_IN_SETTINGS_DOT_PY would need any changing. Am I missing something?
AFAIK, the Django settings are supposed to be immutable. There are multiple reasons for this, the most obvious being that Django is not aware of the server's execution model (prefork / multi-threaded).
Also, you can't load the settings themselves from a Django model because the settings need to be loaded before you can use anything in the ORM.
So, basically, you have two solutions:
you can bootstrap the settings from the database by using any lower-level database access mechanism to load them; or
you can just define your settings in some other model and fetch them directly when you need them.
The first is an incredible hack and I don't suggest it. The second is much more direct and cleaner, but requires you to change your usual habits (from django.conf import settings).
The second approach is probably what's implemented by the 3rd-party apps you linked to.
From Django 1.8 docs:
You shouldn’t alter settings in your applications at runtime.
DATABASES is a dict. So you can manipulate how a dictionary:
import django.conf as conf
conf.settings.DATABASES['default']['NAME'] = 'novo_banco'
Take a look: https://bitbucket.org/bkroeze/django-livesettings
*Django-Livesettings is a project split from the Satchmo Project_. It provides the ability to configure settings via an admin interface, rather than by editing "settings.py".*
Maybe it can be helpful for you.
Honestly I get more Django when I analyze his code. In version 1.4.5 did it (following the module below):
myproject\manage.py
django\core\management__init__.py ## method - execute_manager
django\conf__init__.py ## class - LazySettings; attr - _wrapped
django\utils\functional.py ## class LazyObject; important method -
new_method_proxy
Functional option, but it has its risks. In the python "_" considers the attribute as protected.
from django.conf import settings
settings._wrapped.INSTALLED_APPS = () ## *really work*
In the following project: https://github.com/alexsilva/DJPlugins
you can see this variable being modified at runtime. the idea of the project is already working.
You can use recomended .configure() method of settings module:
from django.conf import settings
settings.configure(DEBUG=True)
settings module has additional handy features. Check docs.
You cannot directly modify the settings.py file
For example:
If u want change the database at runtime, you should Separate the configuration of the database
# Projecr_name/user_database.py
user_database = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'name',
'USER': 'admin',
'PASSWORD': '111111',
'HOST': '127.0.0.1',
'PORT': '3306'
},
'user_db': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'user1',
'USER': 'admin',
'PASSWORD': '22222',
'HOST': '127.0.0.1',
'PORT': '3306'
}
}
# Projecr_name/settings.py
from .user_database import user_database
...
DATABASES = user_database
...
Call in your logical view
# view.py
from ../Projecr_name/user_database import user_database
class Some(...):
def Some(request):
user_database['user_db']['NAME']='user2'
then you can change any setting at runtime use this way
If you have a variable in the settings.py and you want it to change at any time there is 2 ways the
First one is to make a table in database then make for it a serializer then make a view set and any time you want to get it you could send a http request to this viewset then retrieve the data
Second one is to use caching it is so fast and it is familiar to localStorage in angular