In my work we want to run a server with multiple databases. The databases switching should occur when you acces a url like http://myapp.webpage.com or http://other.webpage.com. We want to run only one server instance and at the moment of the HTTP request switch the database and return the corresponding response.
We've been looking for a mantainable and 'Django-friendly' solution. In our investigation we have found possible ways to do this, but we have not enough information about.
Option 1: Django middleware
The django middleware runs each time the server receive a HTTP request.
Making a database switch here could be the best option but using django database routers as far as I know only allow to change the database for a model or group or models.
Another option is to set a django model manager instance in the middleware and force all models to re-assign the objects attribute from an added attribute in the custom middleware.
My last option is to create a new attribute in the request object received by the middleware that return the database alias from settings.py and in each model query use the using method.
Option 2: Class-based View Mixin
Create a mixin that use the past three options, but I the mixin must be set in ALL the Class-based views. If a programmer forget to set the mixin and it comes to a production server, the data could be (or stop being) in the right database, and I don't wanna take the risk.
Option 3: Changing the database settings in runtime
This option works but Is not recommended and is too risky.
UPDATE:
How this works?
middlewares.py
import django.conf as conf
import os.path
class SelectDB(object):
def process_request(self, request):
print request.META['HTTP_REFERER']
file_database = open("booklog/database.txt", "r")
database = file_database.read(10)
file_database.close()
if database != 'default':
conf.settings.DATABASES['default']['NAME'] = database
Any information that help us to solve will be greatly appreciated.
Answer (it worked for me)
The question was already answered here, in stackoverflow. I'd love this functionality were in django. It was a bit hard to find the way to make this possible.
I think that is important to comment the great work that Wilduck made with the django plugin django-dynamic-db-router, it's a great plugin that makes possible this operation in (a bit) different way.
Thanks a lot to #JL Peyret and #ire_and_curses.
And as an answer to #ire_and_curses. At least in this moment, in the project I'm working it's what we need. In previous projects we needed a similar behavior and made one server per instance was terrible to mantain and update each server, even automating the process.
Related
The age-old question: How can I automatically save last_modifed_user in django models?
I found in several places this general process how to do it using thread local. I'm hesitant to simply implement it that way because I'm not entirely sure of the consequences it has and because all these posts are old.
Is using thread local still the "recommended" way of doing this in django 3? Or does django3 have a better options of doing it?
No, this hasn't changed. Simply because separation of concern is an architectural principle of MVC (model-view-controller), which is also how Django (model-view-template) and most web frameworks with ORM are architected. Models know nothing about the request, it's not available (and in many cases there isn't a request at all when a model is saved, think of management commands or regular tasks running in the background).
The alternative to thread local is to make sure you implement it yourself in the controller layer (view layer in Django):
Create a view mixin that you can mix with all the generic views that use the ModelFormMixin to save the user into the model (ModelFormMixin.form_valid()). Or combine it with a form mixin where the user is passed to the form (FormMixin.get_form_kwargs()) and saved when the form is saved (ModelForm.save()).
Create a ModelAdmin mixin that does the same when saving a model in the django admin site.
This of course means someone on your team may forget to do it when creating new views and forms. The link you posted contains an answer as to the advantages and disadvantages of using thread local.
I have written one Django cloud based app. This app will have multiple user and for them multiple database, so that their data should be separate and they can save only to same database.
1) How can we implement it
2) How to automatically one user from login page to assign the database to write on it.
I don't have a complete answer, since you do not give a lot of detail. But here are a couple ots that f hinDjango supports custom database router implementations. A database router is a class that helps django decide which database to use for a particular model. Unfortunately I don't think this mechanism is granular enough for your needs. You can also specify the database to use in your code by using using(name) queryset method and save(using=name) form of save() method for instances. Of course this also means that some features of Django are going to be unvailable to you, since you cannot always expect to have a user. Look at the docs here for more info
https://docs.djangoproject.com/en/dev/topics/db/multi-db/
I have a weird bug with django sessions in my app: some times (about 10 times for ~20000 per day) session information for user is erased. I traced it via log files: at page A there is information for user's session, after it he submits the form and at the next page his session is empty. I tried two types of storage: memcached+db and db only and this problem is for both of them. I tried to reproduce these scenarios, but all works as expected, as I said, it happens very rare. I also checked that this problem exists for different users, and for them is doesn't reproduce each time. I don't have any ideas how to catch the root cause and I don't know what else post here as a description. If someone has any ideas, please let me know. If it is important, I'm running my app with django 1.2 + FastCGI.
Thanks!
UPD: I checked and see that session key from uses is not changed during two sequential requests, at first request there is an actual session state, and at second session variables are relaced with empty.
As a way to debug this problem, I would subclass the standard Django session middleware (or whatever you're currently using):
django.contrib.sessions.middleware.SessionMiddleware
and wrap process_request and (probably more importantly) process_response in some extra logging. Then install your subclassed session middleware in the MIDDLEWARE_CLASSES, rather than the stock Django one.
You could also validate that session.save() has actually committed its changes by attempting to read it back. It could be that the problem lies in session-state serialisation, and it's failing on a particular key or value that you're attempting to store.
None of this will fix your problem, but it might help you to establish what's going on.
As #Steve Mayne mentioned, it would be good to do some logging on the sessions middleware and sessions model save method. That's something I'd start with.
In addition I'd like to say that this could be a database related issue, especially if you're using MySQL database backend for sessions. You can check the log for database locks and other concurrency issues. I had to deal with similar issues before and the solution is clear: optimization and additional performance.
If you have some specific application middleware, you can check for functionality that interferes with Django sessions. Such parallel operations can cause problems, if not implemented properly.
Another thing I would do is to upgrade to the latest stable release of Django and migrate to a mod_wsgi setup.
I'm running multiple sites from a single django instance, and expect the framework to eventually serve several hundred sites from one or several installations.
I need to patch some django settings based on the current request. I've written some middleware to monkey patch the settings, but I need these settings to be patched before the middleware gets invoked because other apps aren't taking the monkey-patched changes (i.e. apps get run then middleware gets run so the apps don't use the monkey-patched settings).
I should also add this is mainly for the benefit of third-party apps that I haven't written, so I don't want to go round adding decorators or anything like that because that would mess up my upgrade path.
So:
How can I get access to the current request in an app's init.py file?
Will an app's init.py get called once per request or only once? If it's only once, how else could I do this so I can manipulate the settings once per request?
Is it safe to do this kind of monkey patching? I know it makes code a bit more opaque, but I don't want to use different wsgi files per site because I want to allow users to edit some of these settings and have my monkey patching come from the database.
Is there a better solution that would allow certain settings to be stored in the database?
This module - django-tupperware does what you are asking about: https://bitbucket.org/jiaaro/django-tupperware/
Give it a try.
Never ever ever change settings on the fly. You cannot predict how the application may one day be deployed, and in most of them the project will fail in.. interesting ways.
If you really want to have hundreds of sites which will be dynamically reconfigured all the time, you might try to Djangos django/conf/__init__.py which is used to supply the settings to the rest of the framework. You might be able to alter it in a way it depends on the request and it's Host field. Or you'll get many interesting failures at the very least.
My solution to this problem is to give each request, ALL the settings and update them on the fly with a middleware, how do i do this?
Is rather simple:
This is the middleware that does it all
from django.conf import settings
class DumpObject: pass
class Settings(object):
def process_request(self,request):
request.settings = DumpObject()
for setting in dir(settings):
if not setting.startswith('__'):
setattr(request.settings, setting, getattr(settings,setting))
The DumpObject is just there so i can use the object.property notation, it could have been a dictionary, but i wanted to keep some similarity in the syntax.
This assumes that none of your settings name starts with __ which is a fair assumption.
So if i want to override a particular setting, i don't keep a settings file i do it in this middleware. like this:
class Settings(object):
def process_request(self,request):
request.settings = DumpObject()
for setting in dir(settings):
if not setting.startswith('__'):
setattr(request.settings, setting, getattr(settings,setting))
if 'mydomain' in str(request.host): #this is thanks to django-hosts project
request.settings.GOOGLE_ANALYTICS_ID = '89298393-238'
Of course this doesnt take into account the problem of accessing the settings the old way
from django.conf import settings
settings.GOOGLE_ANALYTICS_ID = 'the value in settings.py'
But it really doesn't matter because you will probably only want to change the settings on the context of having a request object, available.
What would be the best way to port an existing Drupal site to a Django application?
I have around 500 pages (mostly books module) and around 50 blog posts. I'm not using any 3rd party modules.
I would like to keep the current URLS (for SEO purposes) and migrate database to Django. I will create a simple blog application, so migrating blog posts should be ok. What would be the best way to serve 500+ pages with Django? I would like to use Admin to edit/add new pages.
All Django development is similar, and yours will fit the pattern.
Define the Django model for your books and blog posts.
Unit test that model using Django's built-in testing capabilities.
Write some small utilities to load your legacy data into Django. At this point, you'll realize that your Django model isn't perfect. Good. Fix it. Fix the tests. Redo the loads.
Configure the default admin interface to your model. At this point, you'll spend time tweaking the admin interface. You'll realize your data model is wrong. Which is a good thing. Fix your model. Fix your tests. Fix your loads.
Now that your data is correct, you can create templates from your legacy pages.
Create URL mappings and view functions to populate the templates from the data model.
Take the time to get the data model right. It really matters, because everything else is very simple if your data model is solid.
It may be possible to write Django models which work with the legacy database (I've done this in the past; see docs on manage.py inspectdb).
However, I'd follow advice above and design a clean database using Django conventions, and then migrate the data over. I usually write migration scripts which write to the new database through Django and read the old one using the raw Python DB APIs (while it is possible to tie Django to multiple databases simultaneously, too).
I also suggest taking a look at the available blogging apps for Django. If the one included in Pinax suits your need, go ahead and use Pinax as a starting point.
S.Lott answer is still valid after years, I try to complete the analysis with the tools and format to do the job.
There are many Drupal export tools out of there by now but with the very same request I go for Views Datasource choosing JSON as format. This module is very solid and available for the last version of Drupal. The JSON format is very fast in both parsing and encoding and it's easy to read and very Python-friendly (import json).
Using Views Datasource you can create a node view sorted by node id (nid), show a limited number of elements per page, configure a view path, add to it a filter identifier and pass to it the nid to read all elements until you get an empty JSON response.
When importing in Django you have a wide set of tools as well, starting from loaddata to load fixtures. Views Datasource exported JSON but it's not formatted as Django expects fixtures: you can write a custom admin command to do the import, where you can have the full control of the import flow.
You can start your command passing a nid=0 as argument and then let the procedure read, import and then fetch data from the next page passing simply the last nid read in the previous HTTP request. You can even restrict access to the path on view but you need additional configuration on the import side.
Regarding performance, just for example I parsed and imported 15.000+ nodes in less than 10 minutes via a Django 1.8 custom admin command on an 8 core / 8 GB Linux virtual machine and PostgreSQL as DBMS, logging success and error information into a custom model for each node.
These are the basics for import/export between these two platform, for detailed information I described all the major steps for export from Drupal and then import to Django in this guide.