I have a project with existing class model as
class Disability(models.Model):
child = models.ForeignKey(Child,verbose_name=_("Child"))
But with the recent architecture change i have to modify it as
class Disability(models.Model):
child = models.ManyToManyField(Child,verbose_name=_("Child"))
now for this new change .. ( even i have to modify the existing database for the new one )
i guess data migration is the best way to do it rather than doing it manually .
i refered this online doc
http://south.readthedocs.org/en/latest/commands.html#commands-datamigration
but it has very less about data migration . and more about schema migration .
question 1 . if i do the schema migration will this make me loose all me previous data belonging to that old model.
question 2 . Even i am tring for schema migartion it is asking this ...
(rats)rats#Inspiron:~/profoundis/kenyakids$ ./manage.py schemamigration web --auto
? The field 'Disability.child' does not have a default specified, yet is NOT NULL.
? Since you are removing this field, you MUST specify a default
? value to use for existing rows. Would you like to:
? 1. Quit now, and add a default to the field in models.py
? 2. Specify a one-off value to use for existing columns now
? 3. Disable the backwards migration by raising an exception.
? Please select a choice: 1
Can anyone Explain the concept and difference between schema and data migration and how this can be achieved separately .
Schema and data migrations are not different options you can take to modify your table structure. They are completely different things. Of course, data migrations are fully described in the South docs.
Here a data migration will not help you, because you need to modify your schema. And the whole point of South and other migration systems is that they allow you to do that without losing data.
South will try to do a transaction by moving your table data to a temporary table (I could be wrong there), then restructure the table and try to add in the origin data to the new strucutre. Like this:
old_table -> clone -> tmp_table
old_table ->restructure
tmp_table.data -> table
South will look at the field types. If there is big changes it will ask what to do. For example chaning a text field to a int field would be very hard to convert :)
When you remove fields you may still want to be able to convert back to an old structure, so south will need some default data to be able to create a table with the old structure.
Moving data is always an issue since you may change table structure and field type. For example how would you manually deal with data from a Char(max_length=100) to a Char(max_length=50)?
Best suggestion is to keep good backups.
Also take advantage of djangos fixtures. You can save fixtures for different datastructures along with south migrations.
South will load initial_data files in the same way as syncdb, but it
loads them at the end of every successful migration process
http://south.readthedocs.org/en/latest/commands.html#initial-data-and-post-syncdb
Related
what is the best way to fix this kind of DataBase Errors without having to delete my db and migration files and starting to enter data from scratch?
django.db.utils.IntegrityError: The row in table 'store_product_category' with primary key '1' has an invalid foreign key: store_product_category.category_id contains a value '1' that does not have a corresponding value in store_category.id.
while inspection the sqlit DB i observe that there is a mismatch in the IDs of the store_product_category and store_category.id.
is there anyway i can modify the id directly on the DB, i dont want to start deleting database files and migrations
If I've understood right:
The model StoreProductCategory has a FK - category, linking to a model StoreCategory.
You have a SPC record with category == 1 but no record in StoreCategory with this ID?
If so, the fix is reasonably simple.
Enter the DB shell using python manage.py dbshell and run an SQL INSERT command to add the appropriate record.
Change your model for StoreProductCategory and set on_delete for that FK. I would suggest maybe PROTECT might be appropriate here, but it's up to you - just make sure it's something that will keep things consistent.
If (2) is already done, I do question how this happened in the first place - that would kind of indicate somebody has messed directly with the DB. You may want to investigate who has access and what gets done there.
I use Django 2.2 linked to PostgreSQL and would like to optimise my database queries.
Given the following simplified model:
class Person(model.Models):
name = models.CharField()
age = models.Integerfield()
on which I have to do the following query, say,
Person.objects.filter(age__gt=20, age__lt=30).order_by('name')
What would be the best way to define the index in the model Meta field so as to optimise the query?
Which of these four options would be best?
class Meta
indexes = [models.Index(fields=['age','name']),
models.Index(fields=['name','age']),
models.Index(fields=['name']),
models.Index(fields=['age'])]
Is it, for example, possible to prevent sorting when the query is done? Thank you.
This is really a postgres question, as much as a Django question, right?
I think there is a good chance that creating an index on your sort field will help with performance. But there are a lot of caveats and if it's really important to you, you might want to do some testing focused on Postgres (ie, just run some queries in psql and see what happens). Some caveats include:
it might depend on which type of index is created for you by Django
Postgres, of course, does not always use index anyway when running a query but it should if you've got the right one and the right query (and if there is enough data in the table to justify loading the index)
it might matter how your SELECT is formatted by Django
I suggest you create your model and specify that you want the index. Then use Django Debug Toolbar to find out what SELECT query is really getting run. Then, open a dbshell with manage.py dbshell (aka psql) and run ANALYZE with that same select. Assuming you can interpret the output, you will see for yourself whether your index is coming in to play. Paste the ANALYZE output here, if you like.
According to this Postgres documentation ORDER BY can be assisted by a btree index. The b-tree type of index is what Django will create for you by default.
So, why don't you try this:
class Meta:
indexes = [models.Index(fields=['age', 'name'])]
Then go run an EXPLAIN ANALYZE in dbshell and see whether it worked.
# You should apply indexing on age, because you are searching for 'age' column data
indexes = [
models.Index(fields=['age'])
]
I have a set up (Django 1.11) with several apps including OOK, EEK, and others irrelevant ones. I want to delete all the data for OOK while leaving EEK (and the rest) untouched. Ideally, I want all the primary keys to be reset as well so the first new OOK model will get 1 and so on…
Is this possible?
All I can find is reset and sqlclear which are both deprecated. flush removed all data from the database and thus not what I want
I do release that this is an odd thing to do, but this is the hand given to me…
I think you can achieve this behaviour dropping all the tables of that <app> and then migrating only that <app>. That way you'll reset the tables of the <app>. In django 1.7+ you can do:
$ python manage.py migrate OOK zero //This command unapply your migrations
$ python manage.py migrate OOK
https://docs.djangoproject.com/en/2.0/ref/django-admin/#django-admin-migrate
If you are allowed to replace the db, you could export the data you need to a fixture, then do some clever text processing in the json that is in there, say by finding all ID fields and replacing them from 1. Then reimport the result into a clean db?
The ids are autoincremented by postgresql, according to this answer you can reset the index sequence, but not even sure it can go back to 1.
But really what's the point of resetting the indexes?
This might not be possible with django. However, it is doable with raw SQL:
SET FOREIGN_KEY_CHECKS = 0;
TRUNCATE OOK_table1;
TRUNCATE OOK_table2;
[…]
SET FOREIGN_KEY_CHECKS = 1;
⚠ Do take a backup of your database before doing that!
While using Django 1.7 migrations, I came across a migration that worked in development, but not in production:
ValueError: Found wrong number (0) of constraints for table_name(a, b, c, d)
This is caused by an AlterUniqueTogether rule:
migrations.AlterUniqueTogether(
name='table_name',
unique_together=set([('a', 'b')]),
)
Reading up on bugs and such in the Django bug DB it seems to be about the existing unique_together in the db not matching the migration history.
How can I work around this error and finish my migrations?
(Postgres and MySQL Answer)
If you look at your actual table (use \d table_name) and look at the indexes, you'll find an entry for your unique constraint. This is what Django is trying to find and drop. But it can't find an exact match.
For example,
"table_name_...6cf2a9c6e98cbd0d_uniq" UNIQUE CONSTRAINT, btree (d, a, b, c)
In my case, the order of the keys (d, a, b, c) did not match the constraint it was looking to drop (a, b, c, d).
I went back into my migration history and changed the original AlterUniqueTogether to match the actual order in the database.
The migration then completed successfully.
I had a similar issue come up while I was switching over a CharField to become a ForeignKey. Everything worked with that process, but I was left with Django thinking it still needed to update the unique_together in a new migration. (Even though everything looked correct from inside postgres.) Unfortunately applying this new migration would then give a similar error:
ValueError: Found wrong number (0) of constraints for program(name, funder, payee, payer, location, category)
The fix that ultimately worked for me was to comment out all the previous AlterUniqueTogether operations for that model. The manage.py migrate worked without error after that.
"unique_together in the db not matching the migration history" - Every time an index is altered on a table it checks its previous index and drops it. In your case it is not able to fetch the previous index.
Solution-
1.Either you can generate it manually
2.Or revert to code where previous index is used and migrate.Then finally change to new index in your code and run migration.(django_migration files to be taken care of)
Also worth checking that you only have the expected number of Unique indexes for the table in question.
For example, if your table has multiple Unique indexes, then you should delete them to make sure you have only 1 (or whatever the number of expected Unique indexes is) pre-migration index present.
To check how many Unique indexes are there for a given table in PostgreSQL:
SELECT *
FROM information_schema.table_constraints AS c
WHERE
c.table_name = '<table_name>'
and c.constraint_type = 'UNIQUE'
Just in case someone runs into this and the previous answers haven't solved, In my case the issue was that when I modified the unique together constraint, the migration was attempted but the data didn't allow it (because of a more restrictive unique together constraint). However, the migration managed to delete the unique together constraint from the table leaving it in an inconsistent state. I had to migrate back to zero and re-apply the migration without data. Then it went through without problems.
In summary, make sure your data will be able to accept the new constraint before you apply the migration.
Find the latest migration file of the respective table, find unique
together, and replace current unique constraints fields.
Migrate database using ./manage.py migrate your_app_name.
Revert or undo the previous migrations file.
In my case problem was that the previous migration was not present in the table dajsngo_migrations. I added missing entry and then the new migration worked
Someone may get this issue while modifying the unique_together. Basically, the table state is not consistent with the migrations. You may need to add the previous constraints manually using MySQL shell.
incase one is using migrate with django and there is no data in the database, then drop the database and the do again python manage.py migrate`
I have a django model in use on a production application and I need to change the name and data type of the field with zero downtime to the site. So here is what I was planning:
1) Create the new field in the database that will replace the original field
2) Everytime an instance of the Model is loaded, convert the data form the original field and store it into the new field, then save the object (only save object if new field is empty)
3) Over time the original field can be removed once every object has a non-blank new field
What method can I attach too for the 2nd step?
Won't you have to change your business logic (and perhaps templates) first to accomodate the new fieldname?
Unless stuff gets assigned to the field in question at dozens of places in your code, you could (after creation of the field in the database)
1) adapt the code to recognize the old (read) and the new field(name)s (write).
2) change the data in the database from old to new field via locking / .update() call, etc.
3) remove the old field(name) from the model/views/templates completely
Without downtime, I don't see how users of your site will not suffer getting "old" values for a few seconds (depending on how many rows are in the table, how costly the recalc to the new datatype, etc.).
Sounds complex, and effects a lot of production code.
Are you trying to avoid doing this in bulk because of downtime? What volume of data are you working with?
Have you looked at any Django migration tools that are out there. South is a very popular one:
http://south.aeracode.org/
As you seemingly can't afford ANY downtime whatsoever (I wouldn't want your job!!!) you probably don't want to risk overriding the model's constructor method. What you could try instead is catching the post init signal...
https://docs.djangoproject.com/en/1.0/ref/signals/#django.db.models.signals.post_init