Emptying a table and filling with fixtures - django

I'm working on a big project (tons of migrations already exist, etc...) and I'm at a point in which I must empty an entire table and then fill it again with ~10 elements. What is the Django approach to this situation? Fixtures? RunPython?

Deleting data from tables (Django)
A default Django-generated migration has nothing to do with populating your table. A default migration changes your database layout, e.g. creates or deletes tables, adds columns, changes parameters of columns etc. (Read until the end on how to use manual migrations to delete data from table!)
Deleting data once
What you want to do is delete entries in a table and not delete the whole table. Of course, you could remove the table from your models.py and then migrate which would delete the table (if no errors, read next) but that might result in unwanted behaviour and errors (e.g. other models have ForeignKeys to this table which would probably prevent you from deleting the table). You have two options:
Manually connect to database and run
DELETE * FROM your_table;
Use Python to do the job for you. You can open Django shell by executing python manage.py shell. Then you have to import your model and run .delete(). That would look like this:
$ python manage.py shell
# We are in Django Python shell now...
>> from app.models import Model_to_empty
>> Model_to_empty.objects.all().delete()
Deleting data from tables with manual migration files
If you want to create it as migration then you can write a migration file yourself. To make sure everything is smooth, run
python manage.py makemigrations
python manage.py migrate
first, to migrate any changes that could possible be done in between. Now, create your fake migration file like this:
If your last migration was number 0180, name your file something like 0181_manual_deletion_through_migration.py and put it in app/migrations where app is the app that contains the model that needs to be emptied and refilled.
You can use migrations.RunSQL class in your migrations which will execute statement given as argument while migrating.
Example migration file taken from one of my projects is:
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('beer', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='beer',
name='beer_type',
field=models.CharField(default=0, max_length=30),
preserve_default=False,
),
]
Let's break it down:
dependencies = [
('beer', '0001_initial'),
]
This describes the previous migration that altered the model. 'beer' is the name of the app, '0001_initial' is previous migration. Set this to name of model you want to delete entries from and the name of migration should be the last migration.
operations = [
migrations.AddField(
model_name='beer',
name='beer_type',
field=models.CharField(default=0, max_length=30),
preserve_default=False,
),
]
Inside of operations comes what needs to be done. In my example, it was adding a field, thus migrations.AddField. Remember I told you about migrations.RunSQL? Well, we can use it here like this:
operations = [
migrations.RunSQL("DELETE * FROM your_model;"),
# run SQL statements to populate your model again.
]
where instead of a comment you put the SQL statements that will populate the table with entries you want.
When you finish editing the fake migration file, just execute python manage.py migrate (NO python manage.py makemigrations!!).

Related

Can't add new field in the django model

Django 2.0
I've created a model in Blog app,
class Category(models.Model):
field1 = models....
field2 = models....
field3 = models....
and after some time I want to add a new field in that model.
class Category(models.Model):
field1 = models....
cover_pic = .models....
field2 = models....
field3 = models....
I followed this answer. But it gives me the following error.
django.db.utils.OperationalError: (1054, "Unknown column 'blog_category.cover_pic' in 'field list'")
That answer is from 2014 and for an old Django version.
The workflow for adding new fields is simple:
Write code for the field in your model.
Run command manage.py makemigrations.
Run command manage.py migrate.
This is all in the documentation.
Since, you already followed that answer and have run the manage.py --fake command, you have messed up your db state a little bit.
To recover, do this:
Go to your "blog" app's "migration" folder.
Look at the names of the migration files. They should look like 0001_category_field1.py, 0002_category_cover_pic.py.
The faked migration will be the one with name something like 0002_category_cover_pic.py. Note its number, like 0002.
Now you'll have to move back to the previously applied migration. So, if the faked migration number is 0002, the previously applied migration will be 0001.
Now run this command - manage.py migrate --fake myapp 0001. Note the number of the migration file. You'll have to fake another migration back to the previously applied migration file.
Now run command - manage.py migrate myapp.
This should fix your problem.
If you're problem is not solved yet and you don't have important data in your database, I would suggest start your database from fresh. Delete all your database tables, then delete all the migration files inside your app_name/migration folders. Now run the two commands and start developing.
python manage.py makemigrations
python manage.py migrate
Now you will have a fresh database and you are good to go. From next time try to follow the way mentioned in the above comment.
If you are adding a new field in the Django model and after deploying to any environment , you are getting error while accessing the field.
Error:
"1054, Unknown column"
Although i also was not able to figure out how to resolve it but i came up with a work around.
I created a column manually in the DB .
I added the same field in the model.
Again tried to access the field and it worked like a charm.
I hope it helps your case.

Django migrations using RunPython to commit changes

I want to alter a foreign key in one of my models that can currently have NULL values to not be nullable.
I removed the null=True from my field and ran makemigrations
Because I'm an altering a table that already has rows which contain NULL values in that field I am asked to provide a one-off value right away or edit the migration file and add a RunPython operation.
My RunPython operation is listed BEFORE the AlterField operation and does the required update for this field so it doesn't contain NULL values (only rows who already contain a NULL value).
But, the migration still fails with this error:
django.db.utils.OperationalError: cannot ALTER TABLE "my_app_site" because it has pending trigger events
Here's my code:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import models, migrations
def add_default_template(apps, schema_editor):
Template = apps.get_model("my_app", "Template")
Site = apps.get_model("my_app", "Site")
accept_reject_template = Template.objects.get(name="Accept/Reject")
Site.objects.filter(template=None).update(template=accept_reject_template)
class Migration(migrations.Migration):
dependencies = [
('my_app', '0021_auto_20150210_1008'),
]
operations = [
migrations.RunPython(add_default_template),
migrations.AlterField(
model_name='site',
name='template',
field=models.ForeignKey(to='my_app.Template'),
preserve_default=False,
),
]
If I understand correctly this error may occur when a field is altered to be not-nullable but the field contains null values.
In that case, the only reason I can think of why this happens is because the RunPython operation transaction didn't "commit" the changes in the database before running the AlterField.
If this is indeed the reason - how can I make sure the changes reflect in the database?
If not - what can be the reason for the error?
Thanks!
This happens because Django creates constraints as DEFERRABLE INITIALLY DEFERRED:
ALTER TABLE my_app_site
ADD CONSTRAINT "[constraint_name]"
FOREIGN KEY (template_id)
REFERENCES my_app_template(id)
DEFERRABLE INITIALLY DEFERRED;
This tells PostgreSQL that the foreign key does not need to be checked right after every command, but can be deferred until the end of transactions.
So, when a transaction modifies content and structure, the constraints are checked on parallel with the structure changes, or the checks are scheduled to be done after altering the structure. Both of these states are bad and the database will abort the transaction instead of making any assumptions.
You can instruct PostgreSQL to check constraints immediately in the current transaction by calling SET CONSTRAINTS ALL IMMEDIATE, so structure changes won't be a problem (refer to SET CONSTRAINTS documentation). Your migration should look like this:
operations = [
migrations.RunSQL('SET CONSTRAINTS ALL IMMEDIATE',
reverse_sql=migrations.RunSQL.noop),
# ... the actual migration operations here ...
migrations.RunSQL(migrations.RunSQL.noop,
reverse_sql='SET CONSTRAINTS ALL IMMEDIATE'),
]
The first operation is for applying (forward) migrations, and the last one is for unapplying (backwards) migrations.
EDIT: Constraint deferring is useful to avoid insertion sorting, specially for self-referencing tables and tables with cyclic dependencies. So be careful when bending Django.
LATE EDIT: on Django 1.7 and newer versions there is a special SeparateDatabaseAndState operation that allows data changes and structure changes on the same migration. Try using this operation before resorting to the "set constraints all immediate" method above. Example:
operations = [
migrations.SeparateDatabaseAndState(database_operations=[
# put your sql, python, whatever data migrations here
],
state_operations=[
# field/model changes goes here
]),
]
Yes, I'd say it's the transaction bounds which are preventing the data change in your migration being committed before the ALTER is run.
I'd do as #danielcorreia says and implement it as two migrations, as it looks like the even the SchemaEditor is bound by transactions, via the the context manager you'd be obliged to use.
Adding null to the field giving you a problem should fix it. In your case the "template" field. Just add null=True to the field. The migrations should than look like this:
class Migration(migrations.Migration):
dependencies = [
('my_app', '0021_auto_20150210_1008'),
]
operations = [
migrations.RunPython(add_default_template),
migrations.AlterField(
model_name='site',
name='template',
field=models.ForeignKey(to='my_app.Template', null=True),
preserve_default=False,
),
]

Django migration involving RenameModel not working

I have been successfully using makemigrations and migrate in Django 1.7 for altering, adding and removing fields. Unfortunately, I cannot get it working when trying to rename an intermediate model. I.e. I have two models A and B, linked through a many-to-many field through model X, and I would like to rename X to Y.
Running manage.py makemigrations does not detect the rename, instead it deletes X and adds Y. But that's not the problem. I replaced Django's autogenerated scripts with:
[ migrations.RenameModel(old_name='X',new_name='Y'),
migrations.AlterField(
model_name='Y',
name='a',
field=models.ForeignKey(related_name=b'Y', to='B'),
)]
This gives me the following error:
ValueError: Lookup failed for model referenced by field b: X
So I guess it's struggling with a 'through' relation that contains the old name of the model. I tried adding a migration command to change that relation, updating it to the new name of the intermediate model, but that didn't help either.

django added a new class in a model

This thing never happened to me before,as i never had to create another class in the model field after doing syncdb already. I am now reviewing one of my past projects and i need to add another class in the models.py file. I have very little understanding of south, its more like a procedural one.
when i do this
./manage.py sql app_name
it showls the new table but when i run the server it throws an operational error 'no such table found'. Am i missing something this whole time?? Is there a way??
according to this
./manage.py sql app_name
just print sql statement for create table.
you can write it in a file
./manage.py sql app_name > command.sql
and feed it to database. for example if use postgresql you can use:
psql -U user db_name < command.sql

Django South - schema and data migration at the same time

Isn't it possible to do something like the following with South in a schemamigration?
def forwards(self, orm):
## CREATION
# Adding model 'Added'
db.create_table(u'something_added', (
(u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
('foo', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['something.Foo'])),
('bar', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['something.Bar'])),
))
db.send_create_signal(u'something', ['Added'])
## DATA
# Create Added for every Foo
for f in orm.Foo.objects.all():
self.prev_orm.Added.objects.create(foo=f, bar=f.bar)
## DELETION
# Deleting field 'Foo.bar'
db.delete_column(u'something_foo', 'bar_id')
See the prev_orm that would allow me to access to f.bar, and do all in one. I find that having to write 3 migrations for that is pretty heavy...
I know this is not the "way to do" but to my mind this would be honestly much cleaner.
Would there be a real problem to do so btw?
I guess your objective is to ensure that deletion does not run before the data-migration. For this you can use the dependency system in South.
You can break the above into three parts:
001_app1_addition_migration (in app 1)
then
001_app2_data_migration (in app 2, where the Foo model belongs)
and then
002_app1_deletion_migration (in app 1) with something like following:
class Migration:
depends_on = (
("app2", "001_app2_data_migration"),
)
def forwards(self):
## DELETION
# Deleting field 'Foo.bar'
db.delete_column(u'something_foo', 'bar_id')
First of all, the orm provided by South is the one that you are migrating to. In other words, it matches the schema after the migration is complete. So you can just write orm.Added instead of self.prev_orm.Added. The other implication of this fact is that you cannot reference foo.bar since it is not present in the final schema.
The way to get around that (and to answer your question) is to skip the ORM and just execute raw SQL directly.
In your case, the create statement that accesses the deleted row would look something like:
cursor.execute('SELECT "id", "bar_id" FROM "something_foo"')
for foo_id, bar_id in cursor.fetchall()
orm.Added.ojbects.create(foo_id=foo_id, bar_id=bar_id)
South migrations are using transaction management.
When doing several migrations at once, the code is similar to:
for migration in migrations:
south.db.db.start_transaction()
try:
migration.forwards(migration.orm)
south.db.db.commit_transaction()
except:
south.db.db.rollback_transaction()
raise
so... while it is not recommended to mix schema and data migrations, once you commit the schema with db.commit_transaction() the tables should be available for you to use. Be mindful to provide a backwards() method that does that correct steps backwards.