After setting up history for a model using django-simple-history, I wanted to run populate_history to populate the history table based on the existing contents of the table. However, other users have already made a number of changes, causing the history table to be partially populated. Running populate_history --auto simply results in message Existing history found, skipping model.
I wish to retain the existing history, but populate history for all records not currently stored in the history. Is there a way to do this?
I ended up writing a modified script based on populate_history. It identified all objects with no historical records, and added them in to the history table. A simple version (no batching) is below.
from django.apps import apps
from simple_history.utils import get_history_manager_for_model, get_history_model_for_model
def populate_model_history(model):
history = get_history_model_for_model(model)
history_manager = get_history_manager_for_model(model)
# Insert historical records for objects without existing history
# NOTE: A better approach would be to do this in batches, as in populate_history.py
for instance in model.objects.exclude(pk__in=history.objects.values_list(model._meta.pk.name)):
history_manager.bulk_history_create([instance], batch_size=1)
model = apps.get_model('app', 'my_model')
populate_model_history(model)
Related
I have an sqlite database located at /home/pi/Desktop/Databsaes/data.db and wish to access it from my models.py script.
To view table contents in a normal command prompt I would execute:
sqlite3
sqlite> .open data.db
sqlite> SELECT * from table1
I have been reading through this official tutorial, but I do not understand how to access my local db and perform the above.
In SQL terms, a QuerySet equates to a SELECT statement,
... but how can I perform something of the sort directly in my models.py script?
Models.py is still untouched:
from __future__ import unicode_literals
from django.db import models
# Create your models here
From what I am gathering, field lookups may be used, which specify arguments for QuerySet methods (such as get()).
EDIT
I followed this tutorial to import the database into Django.
However my question is: Will new data added to this database by a separate process be visible from Django's side after the import?
If you do want to allow Django to manage the table’s lifecycle, you’ll need to change the managed option above to True (or simply remove it because True is its default value).
I don not know whether "manage the table's lifecycle" means to update the database with newer data once this is added.
I think the statement of managed=False might be the main point that confusing you. Here is the description on Model._meta.managed:
If False, no database table creation or deletion operations will be performed for this model. This is useful if the model represents an existing table or a database view that has been created by some other means. This is the only difference when managed=False. All other aspects of model handling are exactly the same as normal. This includes
This means new migrations will not be generated (through makemigrations) for the modification of models schema with managed=False. This implies that you are telling Django, "I'm not going to change these model's schema through Django" (But through other way, maybe through another service).
Note that all we were talking about are just the effect on schema changed, which is nothing to do with your real data. After the link of your database and Django model has been established, just as #DanielRoseman's comments, any data that is there will be visible to Django on each query.
Since according to your statement the question is for newly-added data, the answer should be yes. But, if you were meaning that new tables are created through other service (not through the Django service above), of course you still have to add the corresponding model to Django (with managed=False) then you will be able to access data through Django.
We have django-simple-history set up for our models. Recently a whole bunch of models were mysteriously deleted. This was noticed a few days after the fact, so it would be nice to avoid a full DB backup restore since that would wipe manual changes that happened after the fact.
I cannot find any way to easily restore an model instance, specifically a deleted one. I can query the Historical version of the model and find everything that was deleted. With that I can also observe that all of them had deletions as their last change. I can use the instance property on history - 1 to get the state before deletion but if I try to save that it errors since the model instance was deleted and doesn't exist anymore.
So basically, what is the cleanest way to restore a deleted model instance if I have the Historical record of it with django-simple-history? I would like to retain the history if possible, so I am looking into any solution before totally recreating the objects.
As I understand, the question, it is about restoring a deleted model instance, not the class itself. So Kal's answer does not help here.
To restore a deleted instance, simple history can NOT be used. According to the documentation, simple history can only restore model instances that are still existing.
Since you have a full backup of the database, you can import this database into the django environment, load the old model instance from this backup database and save it to the production database (choose the database in django shell with "using"). See this post.
The best way to avoid such situations is to use the app "reversion". With this django-app, you actually can restore deleted instances. See the documentation.
Do you mean that your model, not just the instances, has been completely deleted? If that’s the case, it probably means some migration removed it.
You could try reverting the migration then restore from the Historical record.
The previous version of your model is stored in the _HISTORICAL<model_name> table. You could do a simple sql insert/select query to copy the data from that table back into the original table. Something like the below would work, but check the correct sql syntax for your database.
insert into app_mymodel (col_a, col_b)
select col_a, col_b from app_HISTORICALmymodel
where id in (1,2,3...)
If model using django-simple-history then,
Data can be restored from the historical model.
We need to filter from the historical model. for eg with some keyword or in some date range in which period the data got deleted accidentally.
For eg. the Model Name is Car, then by default django-simple-history creates a historical model as HistoricalCar.
historical_data = HistoricalCar.objects.filter(history_type='-')
for each in historical_data:
instance = each.instance # This return instance of car.
instance.pk = None # Create New Instance
instance.save()
I am currently developing a server using Flask/SqlAlchemy. It occurs that when an ORM model is not present as a table in the database, it is created by default by SqlAlchemy.
However when an ORM class is changed with for instance an extra column is added, these changes do not get saved in the database. So the extra column will be missing, every time I query. I have to adjust my DB manually every time there is a change in the models that I use.
Is there a better way to apply changes in the models during development? I hardly think manual MySql manipulation is the best solution.
you can proceed as the following:
new_column = Column('new_column', String, default='some_default_value')
new_column.create(my_table, populate_default=True)
you can find more details about sqlalchemy migration in: https://sqlalchemy-migrate.readthedocs.org/en/latest/changeset.html
I have a Django website with a PostgreSQL database. There is a Django app and model for a 'flat' item table with many records being inserted regularly, up to millions of inserts per month. I would like to use these records to automatically populate a star schema of fact and dimension tables (initially also modeled in the Django models.py), in order to efficiently do complex queries on the records, and present data from them on the Django site.
Two main options keep coming up:
1) PostgreSQL Triggers: Configure the database directly to insert the appropriate rows into fact and dimensional tables, based on creation or update of a record, possibly using Python/PL-pgsql and row-level after triggers. Pros: Works with inputs outside Django; might be expected to be more efficient. Cons: Splits business logic to another location; triggering inserts may not be expected by other input sources.
2) Django Signals: Use the Signals feature to do the inserts upon creation or update of a record, with the built-in signal django.db.models.signals.post_save. Pros: easier to build and maintain. Cons: Have to repeat some code or stay inside the Django site/app environment to support new input sources.
Am I correct in thinking that Django's built-in signals are the way to go for maintaining the fact table and the dimension tables? Or is there some other, significant option that is being missed?
I ended up using Django Signals. With a flat table "item_record" containing fields "item" and "description", the code in models.py looks like this:
from django.db.models.signals import post_save
def create_item_record_history(instance, created, **kwargs):
if created:
ItemRecordHistory.objects.create(
title=instance.title,
description=instance.description,
created_at=instance.created_at,
)
post_save.connect(create_item_record_history, sender=ItemRecord)
It is running well for my purposes. Although it's just creating an annotated flat table (new field "created_at"), the same method could be used to build out a star schema.
i'd like to use a view i've created in my database as the source for my django-view.
Is this possible, without using custom sql?
******13/02/09 UPDATE***********
Like many of the answers suggest, you can just make your own view in the database and then use it within the API by defining it in models.py.
some warning though:
manage.py syncdb will not work anymore
the view need the same thing at the start of its name as all the other models(tables) e.g if your app is called "thing" then your view will need to be called thing_$viewname
Just an update for those who'll encounter this question (from Google or whatever else)...
Currently Django has a simple "proper way" to define model without managing database tables:
Options.managed
Defaults to True, meaning Django will create the appropriate database tables in syncdb and remove them as part of a reset management command. That is, Django manages the database tables' lifecycles.
If False, no database table creation or deletion operations will be performed for this model. This is useful if the model represents an existing table or a database view that has been created by some other means. This is the only difference when managed is False. All other aspects of model handling are exactly the same as normal.
Since Django 1.1, you can use Options.managed for that.
For older versions, you can easily define a Model class for a view and use it like your other views. I just tested it using a Sqlite-based app and it seems to work fine. Just make sure to add a primary key field if your view's "primary key" column is not named 'id' and specify the view's name in the Meta options if your view is not called 'app_classname'.
The only problem is that the "syncdb" command will raise an exception since Django will try to create the table. You can prevent that by defining the 'view models' in a separate Python file, different than models.py. This way, Django will not see them when introspecting models.py to determine the models to create for the app and therefor will not attempt to create the table.
I just implemented a model using a view with postgres 9.4 and django 1.8.
I created custom migration classes like this:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('myapp', '0002_previousdependency'),
]
sql = """
create VIEW myapp_myview as
select your view here
"""
operations = [
migrations.RunSQL("drop view if exists myapp_myview;"),
migrations.RunSQL(sql)
]
I wrote the model as I normally would. It works for my purposes.
Note- When I ran makemigrations a new migration file was created for the model, which I manually deleted.
Full disclosure- my view is read only because I am using a view derived from a jsonb data type and have not written an ON UPDATE INSTEAD rule.
We've done this quite extensively in our applications with MySQL to work around the single database limitation of Django. Our application has a couple of databases living in a single MySQL instance. We can achieve cross-database model joins this way as long as we have created views for each table in the "current" database.
As far as inserts/updates into views go, with our use cases, a view is basically a "select * from [db.table];". In other words, we don't do any complex joins or filtering so insert/updates trigger from save() work just fine. If your use case requires such complex joins or extensive filtering, I suspect you won't have any problems for read-only scenarios, but may run into insert/update issues. I think there are some underlying constraints in MySQL that prevent you from updating into views that cross tables, have complex filters, etc.
Anyway, your mileage may vary if you are using a RDBMS other than MySQL, but Django doesn't really care if its sitting on top of a physical table or view. It's going to be the RDBMS that determines whether it actually functions as you expect. As a previous commenter noted, you'll likely be throwing syncdb out the window, although we successfully worked around it with a post-syncdb signal that drops the physical table created by Django and runs our "create view..." command. However, the post-syncdb signal is a bit esoteric in the way it gets triggered, so caveat emptor there as well.
EDIT: Of course by "post-syncdb signal" I mean "post-syncdb listener"
From Django Official Documentation, you could call the view like this:
#import library
from django.db import connection
#Create the cursor
cursor = connection.cursor()
#Write the SQL code
sql_string = 'SELECT * FROM myview'
#Execute the SQL
cursor.execute(sql_string)
result = cursor.fetchall()
Hope it helps ;-)