Multi Databases in django: misplaced tables - django

I have two postgres db in my project, one for every app: app1:Store, app2:Warehouse. Both of them have Order model which django names: store_order & warehouse_order (I am fine with that part);
But after migrations there is a problem, both store_order and warehouse_order got in warehouse_db, also all these django tables I want to be only in store_db:
github repo
settings.py
DATABASES = {
'default': {},
'store': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'postgres',
'USER': 'postgres',
'HOST': 'store_db',
'PORT': 5432,
},
'warehouse': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'postgres',
'USER': 'postgres',
'HOST': 'warehouse_db',
'PORT': 5432,
}
}
DATABASE_ROUTERS = ['warehouse.router.WarehouseRouter', 'store.router.StoreRouter']
warehouse/router.py
class WarehouseRouter:
"""
A router to control operations in warehouse app
"""
def db_for_read(self, model, **hints):
if model._meta.app_label == 'warehouse':
return 'warehouse'
else:
return None
def db_for_write(self, model, **hints):
if model._meta.app_label == 'warehouse':
return 'warehouse'
else:
return None
def allow_migrate(self, db, app_label, model_name=None, **hints):
if app_label == 'warehouse':
return db == 'warehouse'
return None
store/router.py
class StoreRouter:
"""
A router to control operations in store app
"""
def db_for_read(self, model, **hints):
return 'store'
def db_for_write(self, model, **hints):
return 'store'
def allow_migrate(self, db, app_label, model_name=None, **hints):
return True
warehouse/models.py also same as store/models.py except of app_label
class Order(models.Model):
id = models.CharField(max_length=128, null=False, unique=True, primary_key=True, default=uuid.uuid1)
amount = models.IntegerField(null=False)
price = models.FloatField(null=False)
comment = models.TextField(null=True)
created_at = models.DateTimeField(null=False, auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
def __str__(self):
return f'{self.id}: {self.created_at}'
class Meta:
app_label = 'warehouse'
I tried to be as close as possible to this example
and my migrations script
#!/bin/sh -
docker-compose run crestopher python manage.py makemigrations
docker-compose run crestopher python manage.py migrate --database=warehouse
docker-compose run crestopher python manage.py migrate --database=store
I also tried to do makemigrations and migrate with one db and then with other, but result was the same.
EDIT
I just run it again on the next day and got:
I don't understand why. I changed migrations script to:
#!/bin/sh -
docker-compose run crestopher python manage.py makemigrations warehouse
docker-compose run crestopher python manage.py migrate --database=warehouse
docker-compose run crestopher python manage.py makemigrations store
docker-compose run crestopher python manage.py migrate --database=store
but on the first day I run it about 1000 times without effect. I'll try to understand what is going on, also these default django tables still should be only in store db. I'll put an answer here if I'll find one.

Related

how to migrate just specific table in django?

I tried multiple databases in django. So, I set databases like that
# settings.py
DATABASE_ROUTERS = [
'stage1.routers.MultiDBRouter',
]
DATABASE_APPS_MAPPING = {
'stage1': 'stage1',
}
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
},
'stage1': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'stage1',
'HOST': 'localhost',
'USER': 'root',
'PASSWORD': '######',
'PORT': 3306,
'CHARSET': 'utf8mb4',
},
}
# stage1.routers.py
class MultiDBRouter(object):
def __init__(self):
self.model_list = ['stage1']
def db_for_read(self, model, **hints):
if model._meta.app_label in self.model_list:
return model._meta.app_label
return None
def db_for_write(self, model, **hints):
if model._meta.app_label == 'stage1':
return 'stage1'
return None
def allow_relation(self, obj1, obj2, **hints):
if (obj1._meta.app_label in self.model_list or obj2._meta.app_label in self.model_list):
return True
return None
def allow_migrate(self, db, app_label, model_name=None, **hints):
if app_label == 'stage1':
return db == 'stage1'
return None
# stage1.models.py
from django.db import models
class Room(models.Model):
name = models.CharField(max_length=100)
sort = models.CharField(max_length=100)
class Meta:
app_label = "stage1"
I did manage.py migrate --database=stage1 and it worked. But there are something wrong I didn't intend.
I just wanted stage1 database has only one table room. But it has all tables that are basically set like auth_group, django_session ...
How can I do to make only one table room in stage1 database?
Please help me.

Django multiple database foreign keys

I am trying to implement multiple database support for my django (version 1.11) app. For that purpose I have included in my settings.py:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'labrin_dbase',
'USER': 'labrin_admin',
'PASSWORD': 'ndzwwZHv63STuvAF?C_$L#j#*#epZXaX',
'HOST': 'localhost',
'PORT': '5432',
},
'comment': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'labrin_dbase_comments',
'USER': 'labrin_admin_comments',
'PASSWORD': 'adzwaTHv63STuvAF!C_$L#j#*#epZXaY',
'HOST': 'localhost',
'PORT': '5433',
}
}
DATABASE_ROUTERS = [
'labrin_task.comment_router.CommentRouter',
]
And my database router is configured as below:
class CommentRouter(object):
def db_for_read(self, model, **hints):
if model._meta.db_table == 'todo_comment':
return 'comment'
return None
def db_for_write(self, model, **hints):
if model._meta.db_table == 'todo_comment':
return 'comment'
return None
def allow_relation(self, obj1, obj2, **hints):
return True
def allow_migrate(self, db, app_label, model_name=None, **hints):
if model_name == 'comment':
return db == 'comment'
return None
Models in my "todo" app(which is only app in project):
from django.db import models
from django.contrib.auth import get_user_model
UserModel = get_user_model()
class Todo(models.Model):
name = models.CharField(max_length=64)
description = models.TextField()
author = models.ForeignKey(UserModel, on_delete=models.CASCADE)
deadline = models.DateTimeField()
created_at = models.DateTimeField(auto_now_add=True)
class Comment(models.Model):
todo = models.ForeignKey(Todo, on_delete=models.CASCADE)
author = models.ForeignKey(UserModel, on_delete=models.CASCADE)
text = models.CharField(max_length=256)
created_at = models.DateTimeField(auto_now_add=True)
class ShareTodo(models.Model):
todo = models.ForeignKey(Todo, on_delete=models.CASCADE)
with_user = models.ForeignKey(UserModel, on_delete=models.CASCADE)
comment_allowed = models.BooleanField(default=False)
When I remove comment database and DATABASE_ROUTERS from settings.py, my app is working normally. After adding mentioned to settings.py, my app returns an error when I create Comment object. The error says:
Exception inside application: insert or update on table "todo_comment" violates foreign key constraint "todo_comment_author_id_bb272a3e_fk_auth_user_id"
DETAIL: Key (author_id)=(1) is not present in table "auth_user". What am I doing wrong?
Note: I am starting two postgres servers as separate docker containers and after running containers, I run python manage.py migrate and python manage.py migrate --database=comment for making all migrations.
Sorry, but cross-database relations are not possible to recreate in Django. You can find full explanation in Django docs.
Furthermore, you cannot even do cross-database relations in PostgreSQL so even trying to hack it or to achieve it outside of Django won't be possible. Maybe for other database engines it is possible, you can do your own research.

What's the best way to implement DB Router in Django?

I have implemented Automatic DB Routing in Django and using AWS Aurora for Database with replication. I have found a minor replication lag with my database which hampering the flow. Issue occurs let's say when a read queryset is getting executed with 'slave' then while updating value using that queryset showing error something like 'read-only access for that table.' that means for update it should route to master db.
Here is my DB Settings for Multiple DB:
DATABASES = {
'master': {
'ENGINE': 'django.db.backends.mysql',
'STORAGE_ENGINE': 'MyISAM / INNODB / ETC',
'NAME': 'db',
'USER': 'master',
'PASSWORD': 'master',
'HOST': 'localhost',
'PORT': '3306',
},
'slave': {
'ENGINE': 'django.db.backends.mysql',
'STORAGE_ENGINE': 'MyISAM / INNODB / ETC',
'NAME': 'db',
'USER': 'name',
'PASSWORD': 'pass',
'HOST': 'localhost',
'PORT': '3306',
},
'slave2': {
'ENGINE': 'django.db.backends.mysql',
'STORAGE_ENGINE': 'MyISAM / INNODB / ETC',
'NAME': 'db',
'USER': 'name',
'PASSWORD': 'pass',
'HOST': 'localhost',
'PORT': '3306',
}
}
DATABASE_ROUTERS = ['path.to.AuthRouter']
Please provide me the best way to handle multiple db route automatically in django.
"""
DB Router core class which auto selects required database configuration
"""
class AuthRouter:
def db_for_read(self, model, **hints):
"""
Reads go to a replica.
"""
print 'db_for_read'
print model
return 'slave'
def db_for_write(self, model, **hints):
"""
Writes always go to master ie default.
"""
print 'db_for_write'
print model
return 'master'
def allow_relation(self, obj1, obj2, **hints):
"""
Relations between objects are allowed if both objects are
in the default/replica pool.
"""
db_list = ('master', 'slave')
if obj1._state.db in db_list and obj2._state.db in db_list:
return True
return None
def allow_migrate(self, db, app_label, model_name=None, **hints):
"""
All non-auth models end up in this pool.
"""
return True
You can split the read operations and isolate the writing operations for master DB like:
AuthRouter should call master DB since that's the fresh information from users.
class AuthRouter:
def db_for_read(self, model, **hints):
"""
Reads go to a replica.
"""
return 'master'
def db_for_write(self, model, **hints):
"""
Writes always go to master ie default.
"""
return 'master'
def allow_relation(self, obj1, obj2, **hints):
"""
Relations between objects are allowed if both objects are
in the default/replica pool.
"""
db_list = ('master', 'slave')
if obj1._state.db in db_list and obj2._state.db in db_list:
return True
return None
def allow_migrate(self, db, app_label, model_name=None, **hints):
"""
All non-auth models end up in this pool.
"""
return True
class PrimaryReplicaRouter(object):
"""
A router to control all read/write database operations.
"""
def db_for_read(self, model, **hints):
"""
Reads go to a randomly-chosen replica.
"""
return select_rand_db()
def db_for_write(self, model, **hints):
"""
Writes always go to primary.
"""
return 'master'
def allow_relation(self, obj1, obj2, **hints):
"""
Relations between objects are allowed if both objects are
in the primary/replica pool.
"""
return True
def allow_migrate(self, db, app_label, model_name=None, **hints):
"""
All non-auth models end up in this pool.
"""
return True
def select_rand_db():
from numpy.random import choice
"""
this function returns rand db or default if running tests
:return:
"""
return choice(['master', 'slave'], p=[0.5, 0.5])

Django migrations with multiple databases

I have a hard time with creating data migrations. I use two databases for my apps. I configured databases in settings.py and also created a router like in Django docs.
# settings.py
DB_HOST = 'localhost'
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'helios',
'HOST': DB_HOST,
'OPTIONS': {
'read_default_file': join(dirname(__file__), 'default.cnf'),
},
},
'other': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'gala_pol',
'HOST': DB_HOST,
'OPTIONS': {
'read_default_file': join(dirname(__file__), 'other.cnf'),
},
},
DATABASE_APPS_MAPPING = {
'contenttypes': 'default',
'auth': 'default',
'admin': 'default',
'sessions': 'default',
'messages': 'default',
'staticfiles': 'default',
'woodsmen': 'default',
'helios': 'default',
'hush': 'default',
'hunt': 'other',
'meat': 'other',
'beast': 'other',
}
# routers.py
class DatabaseAppsRouter(object):
def db_for_read(self, model, **hints):
if model._meta.app_label in settings.DATABASE_APPS_MAPPING:
return settings.DATABASE_APPS_MAPPING[model._meta.app_label]
return None
def db_for_write(self, model, **hints):
if model._meta.app_label in settings.
return settings.DATABASE_APPS_MAPPING[model._meta.app_label]
return None
def allow_relation(self, obj1, obj2, **hints):
db1 = settings.DATABASE_APPS_MAPPING.get(obj1._meta.app_label)
db2 = settings.DATABASE_APPS_MAPPING.get(obj2._meta.app_label)
if db1 and db2:
return db1 == db2
return None
def allow_migrate(self, db, app_label, model_name=None, **hints):
if db in settings.DATABASE_APPS_MAPPING.values():
return settings.DATABASE_APPS_MAPPING.get(app_label) == db
elif app_label in settings.DATABASE_APPS_MAPPING:
return False
Here is the model and migrations of one of those apps:
# hunt.models.py
class Dish(models.Model):
"""
Investigation case
"""
display_name = models.CharField(max_length=64, unique=True)
department = models.ForeignKey(Kitchen, null=True)
case_type = models.PositiveSmallIntegerField(choices=CASE_TYPE_CHOICES, default=DEF_CASE_TYPE)
created_at = models.DateTimeField(blank=True, null=True)
comment = models.CharField(max_length=256, blank=True, null=True)
class Meta:
verbose_name = 'case'
app_label = 'hunt'
def __unicode__(self):
return (u'%s (%s)' % (self.display_name, self.created_at)).strip()
# hunt.migrations.0001_initial.py
class Migration(migrations.Migration):
app_label = 'hunt'
dependencies = [
]
operations = [
migrations.CreateModel(
name='Dish',
fields=[
('id', models.AutoField(verbose_name='ID', auto_created=True, primary_key=True, serialize=False)),
('display_name', models.CharField(max_length=64, unique=True)),
('case_type', models.PositiveSmallIntegerField(default=0, choices=[(0, 'Unknown'), (1, 'General'), (2, 'Terror'), (3, 'Narco'), (4, 'Fraud'), (5, 'Slavery'), (6, 'Traffic'), (7, 'RICO'), (8, 'War'), (9, 'Cyber'), (20, 'Other')])),
('created_at', models.DateTimeField(null=True, blank=True)),
('comment', models.CharField(max_length=256, null=True, blank=True)),
],
options={
'verbose_name': 'case',
},
),
]
# hunt.migrations.0002_add_hunts.py
def create_initial_hunts(apps, schema_editor):
if settings.DEBUG:
print('\nContent added')
class Migration(migrations.Migration):
dependencies = [
('hunt', '0001_initial'),
]
operations = [
migrations.RunPython(create_initial_hunts, hints={'schema_editor': 'other'}),
]
The problem is:
When i run "migrate" command, only applications that connected to default database are migrated. The migrations in rest of the apps are never run. If I launch migrate for such an app with --database option - it works fine.
How can I specify the database per migration? Isn't the router supposed to manage exactly this? Or I missed something else?
You have to run migrate once for each database, specifying the target with --database. Each time it will consult your router to see which migrations to actually perform on that database.
I'm guessing it was designed this way to favor explicitness over implicitness. For example, your workflow might require you to migrate the different databases at different times.
Note, though, that you won't be able to tell from the output which migrations were actually performed, since:
If allow_migrate() returns False, any migration operations for the model_name will be silently skipped when running migrate on the db.
Using these nice helpers you can run Python/SQL migrations on specific Database
[Helpers]
from django.db.migrations import RunPython, RunSQL
def run_python_specific_db_migration(migration_func, use_db):
"""calls RunPython command only for specific database """
return RunPython(migration_func, hints={'use_db': use_db})
def run_sql_specific_db_migration(sql_commands, use_db):
"""Runs one or list of sql commands only on specific database """
return RunSQL(sql_commands, hints={'use_db': use_db})
# ** Your specific db_helpers for your DB_KEY **
def run_sql_your_db_migration(sql_commands):
return run_sql_specific_db_migration(sql_commands, use_db=DB_KEY)
def run_python_your_db_migration(migration_func):
return run_python_specific_db_migration(migration_func, use_db=DB_KEY)
[Usage]
def data_migration(apps, schema_editor):
...your data migration logic..better to wrap with #atomic...
class Migration(migrations.Migration):
operations = [ run_python_your_db_migration(data_migration) ]

Can I point to 2 different Django apps to 2 different urls?

I have a domain named xyz.com running a django project with a database. I have another Django project which should point to xyz.com/pray with a different database. Is it possible to do so?
yes, 2 things i would look at:
a db router for one of your applications
a urls.py files for each of your apps
project/urls.py:
urlpatterns = patterns('',
...
url(r'^pray/', include('prayapp.urls')), # all of this apps ulrs start with /pray
url(r'^', include('otherapp.urls')), # all of these urls start at the root /
...
)
and from https://docs.djangoproject.com/en/1.3/topics/db/multi-db/#an-example:
db settings:
DATABASES = {
# for the rest of your project
'default': {
'NAME': 'app_data',
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'USER': 'postgres_user',
'PASSWORD': 's3krit'
},
# for your prayapp
'other': {
'NAME': 'user_data',
'ENGINE': 'django.db.backends.mysql',
'USER': 'mysql_user',
'PASSWORD': 'priv4te'
}
}
Custom router:
class PrayAppRouter(object):
"""A router to control all database operations on models in
the prayapp application"""
def db_for_read(self, model, **hints):
"Point all operations on prayapp models to 'other'"
if model._meta.app_label == 'prayapp':
return 'other'
return None
def db_for_write(self, model, **hints):
"Point all operations on prayapp models to 'other'"
if model._meta.app_label == 'prayapp':
return 'other'
return None
def allow_relation(self, obj1, obj2, **hints):
"Allow any relation if a model in prayapp is involved"
if obj1._meta.app_label == 'prayapp' or obj2._meta.app_label == 'prayapp':
return True
return None
def allow_syncdb(self, db, model):
"Make sure the prayapp app only appears on the 'other' db"
if db == 'other':
return model._meta.app_label == 'prayapp'
elif model._meta.app_label == 'prayapp':
return False
return None
add this to your settings.py
DATABASE_ROUTERS = ['path.to.PrayAppRouter',]