Running a database migration with RunPython on a second database fails
python3 manage.py migrate --database=app
The problem is that the apps.get_model method takes the default database which has already the newest migrations.
Does not work:
def copy_cpr_cents_to_euros(apps, schema_editor):
User = apps.get_model('accounting', 'User')
User.objects.filter(...);
Works:
def copy_cpr_cents_to_euros(apps, schema_editor):
User = apps.get_model('accounting', 'User')
User.objects.using('app').filter(...);
Is there a way to use the given database in the migration, so in this case "app" without making expliclitly declaring it, since it should work for both databases?
So something like:
User.objects.using(database_name).filter(...)
schema_editor.connection.alias
contains the string of the current database with which the migration was started.
So each RunPython-migration must use this alias to manually select the right db.
Example:
def copy_cpr_cents_to_euros(apps, schema_editor):
User = apps.get_model('accounting', 'User')
db = schema_editor.connection.alias
User.objects.using('app').using(db).filter(...)
Decorators which can be used for RunPython function to specify against which DB it should be executed [Tested on Django 1.8]
def only_default_db_migration(func):
return only_databases_migration('default')(func)
def only_databases_migration(*db_aliases):
"""Restrict running Data Migrations on wanted databases only"""
def decorate(func):
def run_python_func(apps, schema_editor):
db_alias = schema_editor.connection.alias
if db_alias in db_aliases:
return func(apps, schema_editor)
else:
print(f'Skip RunPython {func.__name__!r} for db with alias {db_alias!r}')
return run_python_func
return decorate
Usage of only_default_db_migration
#only_default_db_migration
def migrate_spam_check_processed_at(apps, schema_editor):
apps.get_model("queues","Queue").objects.all().update(check=F('created'))
class Migration(migrations.Migration):
operations = [
migrations.RunPython(migrate_spam_check_processed_at),
]
Related
Let us suppose that we had a model (example is from the documentation https://docs.djangoproject.com/en/4.1/ref/models/fields/#filefield):
def user_directory_path(instance, filename):
# file will be uploaded to MEDIA_ROOT/user_<id>/<filename>
return 'user_{0}/{1}'.format(instance.user.id, filename)
class MyModel(models.Model):
upload = models.FileField(upload_to=user_directory_path)
Migrations will be something like:
('upload', models.FileField(upload_to=vocabulary_phrases.models.user_directory_path)),
But now you decide to change to a class:
class UploadTo:
def __init__(self, folder, filename_suffix=""):
self.folder = folder
self.filename_suffix = filename_suffix
def _get_filename(self, instance, filename):
_, file_extension = os.path.splitext(filename)
result = str(instance.a_uuid)
if self.filename_suffix:
result += "-{}".format(self.filename_suffix)
result += file_extension
return result
def save_path(self, instance, filename):
tmp_filename = self._get_filename(instance, filename)
result = "{}/{}".format(self.folder.value,
tmp_filename)
return result
class MyModel(models.Model):
upload = UploadTo(folder=UPLOAD_TO.VOCABULARY_FILE_FOLDER, filename_suffix="trololo").save_path
When you try to makemigrations, this code will blow. It will complain like 0002_migration contains user_directory_path, it is absent.
This change in the code not mechanical. I can't imagine that it can be done just by refactor / rename in IDE. Then I can't imagine what the new migration should look like. This means that I will not be able to modify migrations file easily.
I will have to deploy another project next to this one and another database. Delete all migrations, makemigrations, copy created migration and substitute all occurrances of user_directory_path with what is in my clipboard.
Complicated, I would say. And I suspect then it may be error prone.
This was just one example. Django hardcodes something in migrations which I would say is related more to business logics than to the database itself. Let us not discuss whether this example is good or not. It illustrates how something from our code is hardcoded in migrations and Django fails automatically make new migrations.
Could you recommend me what is the best practice of coping with this problem?
A simple solution would be to run
python manage.py makemigrations
while you still have user_directory_path in your file
eg:
import os
from django.db import models
def user_directory_path(instance, filename):
# file will be uploaded to MEDIA_ROOT/user_<id>/<filename>
return "user_{0}/{1}".format(instance.user.id, filename)
class UploadTo:
def __init__(self, folder, filename_suffix=""):
self.folder = folder
self.filename_suffix = filename_suffix
def _get_filename(self, instance, filename):
_, file_extension = os.path.splitext(filename)
result = str(instance.a_uuid)
if self.filename_suffix:
result += "-{}".format(self.filename_suffix)
result += file_extension
return result
def save_path(self, instance, filename):
tmp_filename = self._get_filename(instance, filename)
result = "{}/{}".format(self.folder.value, tmp_filename)
return result
class UPLOAD_TO:
VOCABULARY_FILE_FOLDER = "asd"
class MyModel(models.Model):
upload = UploadTo(
folder=UPLOAD_TO.VOCABULARY_FILE_FOLDER, filename_suffix="trololo"
).save_path
then you can squash migrations and delete the function which will avoid the need to deploy a copy project
Getting following error even after I have cleared all migrations.All was working fine until I tried to primary_key to my Cylinderentry model later on I removed but still it displays "django.db.utils.OperationalError: foreign key mismatch - "Entry_returncylinder" referencing "Entry_cylinderentry" "
when I run py manage.py makemigrations it displays:
You are trying to add a non-nullable field 'id' to cylinderentry without a default; we can't do that (the database needs something to populate existing rows).splays following :
or when I run py manage.py migrate it displays:
django.db.utils.OperationalError: foreign key mismatch - "Entry_returncylinder" referencing "Entry_cylinderentry"
here is all models:
from django.db import models
from django.utils import timezone
from django.urls import reverse
# Create your models here.
class CylinderEntry(models.Model):
stachoice=[
('Fill','fill'),
('Empty','empty')
]
substachoice=[
('Available','availabe'),
]
cylinderId=models.CharField(max_length=50,unique=True)
gasName=models.CharField(max_length=200)
cylinderSize=models.CharField(max_length=30)
Status=models.CharField(max_length=40,choices=stachoice,default='fill')
Availability=models.CharField(max_length=40,choices=substachoice,default="available")
EntryDate=models.DateTimeField(default=timezone.now)
def get_absolute_url(self):
return reverse('cylinderDetail',args=[(self.id)])
def __str__(self):
return self.cylinderId
class IssueCylinder(models.Model):
cylinder=models.OneToOneField('CylinderEntry',on_delete=models.CASCADE)
userName=models.CharField(max_length=60)
issueDate=models.DateTimeField(default=timezone.now)
def save(self,*args,**kwargs):
if not self.pk:
CylinderEntry.objects.filter(cylinderId=self.cylinder.cylinderId).update(Availability=('issued'))
super().save(*args,**kwargs)
def __str__(self):
return self.userName
class ReturnCylinder(models.Model):
rechoice=[
('fill','Fill'),
('empty','Empty')
]
reav=[
('Yes','yes'),
('No','no')
]
cylinder=models.ForeignKey('CylinderEntry',on_delete=models.CASCADE)
user=models.ForeignKey('IssueCylinder',on_delete=models.CASCADE)
status=models.CharField(max_length=20,choices=rechoice)
returnDate=models.DateTimeField(default=timezone.now)
Availability=models.CharField(max_length=5,choices=reav)
def save(self,*args,**kwargs):
if not self.pk:
IssueCylinder.objects.get(userName=self.user.userName).delete()
if self.status=='yes' or self.status=='Yes':
CylinderEntry.objects.get(cylinderId=self.cylinder.cylinderId).update(Availability=('available'))
else:
CylinderEntry.objects.get(cylinderId=self.cylinder.cylinderId).delete()
super().save(*args,**kwargs)
def __str__(self):
return self.cylinder
I don't where I m missing. Help!!
I am trying to write some unittests for my flask application and for some reason my database is always empty.
import os
import unittest
import json
from flask_sqlalchemy import SQLAlchemy
from flaskr import create_app
from models import setup_db, Question, Category, db
class TriviaTestCase(unittest.TestCase):
"""This class represents the trivia test case"""
def setUp(self):
"""Define test variables and initialize app."""
self.app = create_app()
self.client = self.app.test_client
self.database_name = "trivia_test"
self.database_path = "postgres://{}/{}".format('localhost:5432', self.database_name)
setup_db(self.app, self.database_path)
self.question = {
'question': 'is Yaser amazing?',
'answer': 'of course, are you crazy?',
'difficulty': 1 ,
'category': 'all'
}
# binds the app to the current context
with self.app.app_context():
self.db = SQLAlchemy()
self.db.init_app(self.app)
# create all tables
self.db.create_all()
def tearDown(self):
"""Executed after reach test"""
pass
def test_get_questions(self):
res = self.client().get('/questions')
data = json.loads(res.data)
print(data)
self.assertEqual(res.status_code, 200)
self.assertEqual(data['success'],True)
self.assertTrue(data['totalQuestions'])
self.assertTrue(data['all_categories'])
I thought that self.question{} would add a row in my database but it doesn't. I am not sure if the syntax is right or if it should be something else. I am following a class example and I am stumped.
It looks like you're adding data to your test class object but not adding to your database. To demonstrate:
# The following adds a dictionary to `self.question`
self.question = {
'question': 'is Yaser amazing?',
'answer': 'of course, are you crazy?',
'difficulty': 1 ,
'category': 'all'
}
# The following assertion should be true
assert isinstance(self.question, dict)
To add to your database you would need to do the following:
def add_q(self, commit=True):
"""
Unpack the stored dictionary into your db model,
then add to your db.
If you'd like to query this object from the db,
you'll need to commit the session.
--> Toggle this feature with the param `commit`
"""
self.db.session.add(Question(**self.question))
# You can also commit outside of the function execution
# Helpful for batch processing
if commit:
self.db.session.commit()
After doing the above, you should be able to query your database for the newly added question. If you're running this test often, you'll likely want to remove the newly added q. Here's a helpful function for that:
def remove_q(self, commit=True):
"""
Query your db for the previously added q. Then
remove from db.
To commit this change, set commit=True (default)
"""
my_q = self.db.session.query(Question).filter_by(**self.question).first()
# now delete
self.db.session.delete(my_q)
# You can also commit outside of the function execution
# Helpful for batch processing
if commit:
self.db.session.commit()
In my Django project I have a couple of applications, one of them is email_lists and this application does a lot of data handling reading data from the Model Customers. In my production environment I have two databases: default and read-replica. I would like all queries in a particular module to be made against the replica-set database.
I can do that if I explicitly tell the query to do so:
def get_customers(self):
if settings.ENV == 'production':
customers = Customer.objects.using('read-replica').filter()
else:
customers = Customer.objects.filter()
but this module has more than 100 queries to the Customer and other models. I also have queries to relations like:
def get_value(self, customer):
target_sessions = customer.sessions.filter(status='open')
carts = Cart.objects.filter(session__in=target_sessions)
the idea is that I want to avoid writing:
if settings.ENV == 'production':
instance = Model.objects.using('read-replica').filter()
else:
instance = Model.objects.filter()
for every query. There are other places in my project that do need to read from default database so it can't be a global setting. I just need this module or file to read using the replica.
Is this possible in Django, are there any shortcuts ?
Thanks
You can read on django database routers for this, some good examples can be found online as well and they should be straightforward.
--
Another solution would be to modify the Model manager.
from django.db import models
class ReplicaRoutingManager(models.Manager):
def get_queryset(self):
queryset = super().get_queryset(self)
if settings.ENV == 'production':
return queryset.using('read-replica')
return queryset
class Customer(models.Model):
...
objects = models.Manager()
replica_objects = ReplicaRoutingManager()
with this, you can just use the normal Customer.objects.filter and the manager should do the routing.
I still suggest going with the database router solution, and creating a custom logic in the class. But if the manager works for you, its fine.
If you want All the queries in the email_lists app to query read-replica, then a router is the way to go. If you need to query different databases within the same app, then #ibaguio's solution is the way to go. Here's a basic router example similar to what I'm using:
project/database_routers.py
MAP = {'some_app': 'default',
'some_other_app': 'default',
'email_lists': 'read-replica',}
class DatabaseRouter:
def db_for_read(self, model, **hints):
return MAP.get(model._meta.app_label, None)
def db_for_write(self, model, **hints):
return MAP.get(model._meta.app_label, None)
def allow_relation(self, object_1, object_2, **hints):
database_object_1 = MAP.get(object_1._meta.app_label)
database_object_2 = MAP.get(object_2._meta.app_label)
return database_object_1 == database_object_2
def allow_migrate(self, db, app_label, model=None, **hints):
return MAP.get(app_label, None)
In settings.py:
DATABASE_ROUTERS = ['project.database_router.DatabaseRouter',]
It looks like you only want it in production, so I would think you could add it conditionally:
if ENV == 'production':
DATABASE_ROUTERS = ['project.database_router.DatabaseRouter',]
I am using django_fsm to manage state in my model. My model looks like:
from django.db import models,
from django_fsm import FSMField, transition
class MyModel(models.Model):
STATES = (
('pending', _('Pending')),
('active', _('Active'))
)
state = FSMField(choices=STATES, default='pending', protected=True)
#transition(field=state, source='pending', target='active')
def change_state(self):
pass
Should I add self.save() to change_state? Will it be called?
If calling change_state() succeeds without raising an exception, the state field will be changed, but not written to the database.
So for making changes into database you need to call obj.save() explicitly
def change_view(request, model_id):
obj = get_object__or_404(MyModel, pk=model_id)
obj.change_state()
obj.save()
return redirect('/')
You can use a post_transition signal to handle this:
from django_fsm.signals import post_transition
#receiver(post_transition, sender=models.MyModel)
def save_new_workflow_state(sender, instance, name, source, target, **kwargs):
""" Save the new workflow state following successful django_fsm transition. """
if source != target:
instance.save()
This comes from this issue.