I'm trying to sync data between two django installations (production and testing). I'm doing this using ./manage.py dumpdata --natural on production, then ./manage.py loaddata into a freshly syncdb'ed database on testing.
Everything was working fine until I added a new custom permission. The production syncdb loaded this new permission in a different order (with different primary key) than a new syncdb on an empty database does. Consequently, it gets a different ID. So despite using natural keys, when I attempt to load the data, I'm getting this error when the first out-of-order permission object is loaded:
IntegrityError: duplicate key value violates unique constraint "auth_permission_content_type_id_codename_key"
The easiest way I can think of to fix this is to remove all data from every table in the testing installation -- that is, to use syncdb just to create tables, and not to also load initial data. But syncdb doesn't let you skip the initial data/signals step. Short of enumerating every model or table name explicitly, how can I remove all initial data after calling syncdb? Or is there a way to create just the empty tables without using syncdb?
./manage.py flush isn't what I'm after -- it reloads initial data and triggers syncdb signals.
According to the help for flush command (I'm using Django 1.3.1) the SQL that is executed is the same SQL obtained from ./manage.py sqlflush, and then the initial data fixtures is reinstalled.
$ python manage.py help flush
Usage: manage.py flush [options]
Executes ``sqlflush`` on the current database.
To get the same data wiping capabilities minus the fixture loading you can obtain the SQL by calling ./manage.py sqlflush and then execute that SQL using Django's built-in support for executing arbitrary SQL:
from django.core.management import call_command, setup_environ
from your_django_project import settings
setup_environ(settings)
from django.db import connection
from StringIO import StringIO
def main():
# 'call' manage.py flush and capture its outputted sql
command_output = StringIO()
call_command("sqlflush", stdout=command_output)
command_output.seek(0)
flush_sql = command_output.read()
# execute the sql
# from: https://docs.djangoproject.com/en/dev/topics/db/sql/#executing-custom-sql-directly
cursor = connection.cursor()
cursor.execute(flush_sql)
print "db has been reset"
if __name__ == '__main__':
main()
This has the added benefit that you can modify the SQL from ./manage.py sqlflush before execution to avoid wiping tables that you might want to leave intact.
Also, according to the current Django docs, in Django 1.5 a new parameter ./manage.py flush --no-initial-data will reset the data and not load the initial data fixture.
For Django <= 1.4, you can use the reset management command.
./manage.py sqlreset myapp1 myapp2
Related
My project use an SQLite3 DB but now I need to do a migration from SQLite3 to PostgreSQL. I've used DBeaver for do this using the "Import Data" option. At the end of the importation of all the tables from SQLite3 to PostgreSQL I noticed this error when I try to add a new content:
IntegrityError at /engine/kernel/keyconcept/add/
duplicate key value violates unique constraint
"kernel_keyconcept_pkey" DETAIL: Key (id)=(4) already exists.
If I add a new model, into this project, that use the new DB(PostgreSQL) there aren't problems to add contents, but if I use the old models I see the IntegrityError above.
I've do a test with a model that have only two rows: with the first attempt I received a same error but for key id 1, with the second attempt I received a same error but for key id 2. When I've tried for the third time I was be able to add the content and the same think happens for the next attempts.
So I think that the row counter of the every tables after the migrations never start from the last content but from zero. It strange because previous I've do the same migrations but from SQLite3 to SQLite3 whitout problems.
How I can solve this?
You have to reset the sequences used to generate primary keys. There is a management command that prints the necessary SQL: sqlsequencereset
Note that it only prints out the SQL, you have to execute it manually (which you can probably do with DBeaver).
I've solved using dumpdata and loaddata instead of DBeaver and following this examples all work fine.
In my case I need to exclude some apps as auth and contenttypes, then with dumpdata I've used this:
python3 manage.py dumpdata --exclude contenttypes --exclude auth --indent 2 > db.json
To upload tha datas I've used this:
python3 manage.py loaddata db.json
Build: Heroku Python server, Postgresql 10.4, Django 2, wagtail 2.1
I'm trying to essentially destroy and recreate my app DB on Heroku. Here are the steps I've followed:
create db dump (success)
rm all migrations and recreated the 'initial' migrations (success)
run `heroku pg:reset DATABASE` (success)
push new migrations and db dump (success)
run `heroku run python manage.py migrate` (success)
run `heroku run python manage.py flush` (**failed**)
JVsquad$ heroku run python manage.py flush
Running python manage.py flush on ⬢ my_app... up, run.2459 (Hobby)
You have requested a flush of the database.
This will IRREVERSIBLY DESTROY all data currently in the 'my_app_db' database,
and return each table to an empty state.
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: yes
CommandError: Database my_app_db couldn't be flushed. Possible
reasons:
* The database isn't running or isn't configured correctly.
* At least one of the expected database tables doesn't exist.
* The SQL was invalid.
Hint: Look at the output of 'django-admin sqlflush'. That's the SQL this command wasn't able to run.
My 7th step was going to be heroku run python manage.py loaddata db_dump.json but it also failed, because the flush won't work.
HELP PLEASE
If nothing works, you can delete the database form heroku GUI and provision a fresh database. this would solve your immediate problem.
Also, this thread suggests a solution
https://github.com/wagtail/wagtail/issues/1824
I had the same issue after moving from Mysql to PostgresQL.
The issue with TRUNCATE tablename CASCADE was that I couldn't set the allow_cascade=True in the flush method (that is run after each test case method)
I override the Transaction class used in my tests: APITransactionTestCase
class PostgresTestCase(APITransactionTestCase):
"""
Override APITransactionTestCase class so the TRUNCATE table_name CASCADE is enabled
"""
def _fixture_teardown(self):
# Allow TRUNCATE ... CASCADE and dont emit the post_migrate signal
# when flushing only a subset of the apps
for db_name in self._databases_names(include_mirrors=False):
# Flush the database
inhibit_post_migrate = (
self.available_apps is not None or
( # Inhibit the post_migrate signal when using serialized
# rollback to avoid trying to recreate the serialized data.
self.serialized_rollback and
hasattr(connections[db_name], '_test_serialized_contents')
)
)
call_command('flush', verbosity=3, interactive=False,
database=db_name, reset_sequences=False,
allow_cascade=True,
inhibit_post_migrate=True)
Now the flush works in Postgres too.
I run python manage.py makemigrations and I get:
No changes detected
Then, python manage.py migrate and I get:
No migrations to apply.
Then, I try to push the changes to production:
git push heroku master
Everything up-to-date
Then, in production, I repeat the command:
heroku run python manage.py migrate
No migrations to apply.
Just in case, I run makemigrations in production:
heroku run python manage.py makemigrations
No changes detected
WHY then I get a
ProgrammingError at ....
column .... does not exist
"No changes detected" means the database is coherent with the code.
How can I debug this?¡?
I got the same problem (column not exist) but when I try to run migrate not with makemigrations (it is the same issue I believe)
Cause: I removed the migration files and replaced them with single pretending intial migration file 0001 before running the migration for the last change
Solution:
Drop tables involved in that migration of that app (consider a backup workaround if any)
Delete the rows responsible of the migration of that app from the table django_migrations in which migrations are recorded, This is how Django knows which migrations have been applied and which still need to be applied.
And here is how solve this problem:
log in as postgres user (my user is called posgres):
sudo -i -u postgres
Open an sql terminal and connect to your database:
psql -d database_name
List your table and spot the tables related to that app:
\dt
Drop them (consider drop order with relations):
DROP TABLE tablename ;
List migration record, you will see migrations applied classified like so:
id | app | name | applied
--+------+--------+---------+
SELECT * FROM django_migrations;
Delete rows of migrations of that app (you can delete by id or by app, with app don't forget 'quotes'):
DELETE FROM django_migrations WHERE app='yourapp';
log out and run your migrations merely (maybe run makemigrations in your case):
python manage.py migrate --settings=your.settings.module_if_any
Note: it is possible that in your case will not have to drop all the tables of that app and not all the migrations, just the ones of the models causing the problem.
I wish this can help.
Django migrations are recorded in your database under the 'django_migrations' table. This is how Django knows which migrations have been applied and which still need to be applied.
Have a look at django_migrations table in your DB. It may be that something went wrong when your migration was applied. So, delete the row in the table which has the migration file name that is related to that column that 'does not exist'. Then, try to re-run a migration.
Here's what i tried and it worked:
Go and add manually the column to your table
run python manage.py makemigrations
go back drop that column you added
run python manage.py migrate
I had a similar issue - the error message appeared when I clicked on the model on the django-admin site. I solved it by commenting out the field in models.py, then running migrations. Following this I uncommented the field and re ran the migrations. After that the error message disappeared.
My case might be a bit obscure, but if it helps someone, it is worth documenting here.
I was calling a function in one of my migrations, which imported a Model of said migration regularly, i.e.
from myApp.models import ModelX
The only way models should be imported in migrations would be using e.g. RunPython:
def myFunc(apps, schema_editor):
MyModel = apps.get_model('myApp 'MyModel')
and then calling that function like so:
class Migration(migrations.Migration):
operations = [
migrations.RunPython(initialize_mhs, reverse_code=migrations.RunPython.noop),
]
Additionally the original import worked until I modified the model in a later migration, making this error harder to locate.
So, I always run into this sort of problem, so today I decided to try and work it out at the database level. Thing is, I changed a model field name which Django didn't bother reflecting in a migration file. I only found out later when I ran into problems. I later looked at the migration files and discovered there was no migration for that change. But I didn't notice because I made other changes as well, so once I saw a migration file I was happy.
My advice. Create migration for each change one at a time. That way you get to see if it happened or not.
So here's my working through it in MySQL.
open mysql console.
show databases; # see all my dbs. I deleted a few
drop database <db-name>; # if needed
use <db-name>; # the database name for your django project
show tables; # see all tables in the database
DESCRIBE <table-name>; # shows columns in the database
SHOW COLUMNS FROM <db-name>; # same thing as above
ALTER TABLE <table-name> CHANGE <old-column-name> <new-column-name> <col-type>; # now I manually updated my column name
If you're using postgresql, just google the corresponding commands.
The issue was in the Models for me, for some reason Django was adding '_id' to the end of my Foreign Key column. I had to explicitly set the related named to the Foreign Key. Here 'Cards' is the parent table and 'Prices' is the child table.
class Cards(models.Model):
unique_id = models.CharField(primary_key=True, max_length=45)
name = models.CharField(max_length=225)
class Prices(models.Model):
unique_id = models.ForeignKey(Cards, models.DO_NOTHING)
Works after changing to:
class Cards(models.Model):
unique_id = models.CharField(primary_key=True, max_length=45)
name = models.CharField(max_length=225)
class Prices(models.Model):
unique_id = models.ForeignKey(Cards, models.DO_NOTHING, db_column='unique_id')
When I get this error, my extreme way to solve it is to reset my database:
Reset your database
For Postgresql on Heroku:
Heroku > your_app > Resources > database > add-ons > click on your database and open it
For postgresql
settings > Reset database
Delete all files in your_app > migrations > __pycache__ except __init.py__
Delete all files in your_app > migrations except __pycache__ folder and __init.py__
Then run:
python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser
type in to create your superuser, then run:
python manage.py makemigrations
python manage.py migrate
python manage.py
If you are able to inspect your models from your admin section, then it should be all okay now.
Just remove corresponding row migrations for that model in 'django_migrations' model in database.
And re run python manage.py migrate app_name
I tried all these answers with not much luck! What I did to have this problem solved with no harm was to go back through the migration files and find where the actual model was being created for the first time then manually add the field (in the column not being existed error message). Until if you run makemigrations --dry-run you get/see "No changes detected" and that worked. Basically, in my case, I had to carefully take my desired db changes back in time in proper migration file, rather creating a new migration now at the end of migration dependency chain.
Open the latest py file created after running the makemigrations command inside migrations folder of that particular app.
Inside class Migration there is a list attribute called 'operations'.
Remove the particular elements migrations.RemoveField(...).
Save and run python manage.py migrate.
A easier solution to the problem is to make your models exactly like it is in the migration first. and run python manage.py migrate.
Then revert those changes
Run
python manage.py makemigrations
python manage.py migrate
To check for which migrations are applied and which are not, use -:
python manage.py showmigrations
I solved a similar problem by deleting all migrations files (Don't forget to make a backup) and python manage.py makemigrations all of them into one clean file in development and pulling new files on the server. Before then I had dropped existing tables on the PostgreSQL.
I was getting this error for some reason when Django was looking for a column of type ForeignKey named category_id when the actual name in the database was category. I tried every Django solution I could imagine (renaming field, explicitly setting column name, etc.). I didn't want to drop tables or rows as this was a production database. The solution was simply to rename the column manually using SQL. In my case:
ALTER TABLE table_name
RENAME COLUMN category TO category_id;
Make sure you backup your database, ensure this won't break any other applications consuming that particular table, and consider having a fallback column if necessary.
What helped me in the end was simply dropping the database and creating it again as well as deleting all migrations files (including cache). (only removing migrations file didn't work for me at all)
sudo su - postgres
psql
DROP DATABASE 'yourdatabase';
CREATE DATABASE 'yourdatabase';
GRANT ALL PRIVILEGES ON DATABASE 'yourdatabase' to 'yourdjangouser';
then just
python manage.py makemigrations
python manage.py migrate
python manage.py runserver
If you're in development and you make some examples of data that's not important, this step is beneficial for me: just flush your data, make migrations, and migrate:
python manage.py flush
python manage.py makemigrations
python manage.py migrate
After that, you may create a new database from scratch, I hope this information was helpful.
Solved this issue by running
python manage.py migrate
in Heroku Bash shell
According to the documentation here:
https://docs.djangoproject.com/en/1.8/topics/migrations/ it says:
migrate, which is responsible for applying migrations, as well as unapplying and listing their status.
and
makemigrations, which is responsible for creating new migrations based on the changes you have made to your models.
From what I understand, I first do
makemigrations
to create the migration file and then do
migrate
to actually apply the migration?
Do note though that I just began my Django project and I added my app to my "installed_apps" list. After that, I did
python manage.py runserver
and it said
You have unapplied migrations; your app may not work properly until they are applied. Run 'python manage.py migrate' to apply them.
It didn't mention anything about running makemigrations.
According the Polls tutorial:
python manage.py makemigrations <app>: Create the migrations (generate the SQL commands).
python manage.py migrate: Run the migrations (execute the SQL commands).
As Django's documentation says Migrations are Django’s way of propagating changes you make to your models (adding a field, deleting a model, etc.) into your database schema.
makemigrations basically generates the SQL commands for preinstalled apps (which can be viewed in installed apps in settings.py) and your newly created apps' model which you add in installed apps.It does not execute those commands in your database file. So tables doesn't created after makemigrations.
After applying makemigrations you can see those SQL commands with sqlmigrate which shows all the SQL commands which has been generated by makemigrations.
migrate executes those SQL commands in database file.So after executing migrate all the tables of your installed apps are created in your database file.
You can conform this by installing sqlite browser and opening db.sqlite3 you can see all the tables appears in the database file after executing migrate command.
As we know Django is an ORM (Object Relational Mapping). When we use the command:
python manage.py makemigrations [app_name]
It will generate the sql command to create the table corresponding to each class you made in models.py file.
then the command:
python manage.py migrate [app_name]
will create the table in database using the commands which have been generated by makemigrations.
For example, if we make a model class-
from django.db import models
class Person(models.Model):
first_name = models.CharField(max_length=30)
last_name = models.CharField(max_length=30)
The corresponding sql command after using makemigrations will be
CREATE TABLE myapp_person (
"id" serial NOT NULL PRIMARY KEY,
"first_name" varchar(30) NOT NULL,
"last_name" varchar(30) NOT NULL
);
and using above command, table will be created in the database when we use migrate.
You should run the command -migrate- after adding a new app under the INSTALLED APPS section in the settings.py file in order to synchronize the database state with your current set of models. Assuming you've already modified the models.py file.
When you run -makemigrations- it packages up changes to your model into individual migration files.
Normally you would first run makemigrations and then migrate.
See documentation on Django Models
It is necessary to run both the commands to complete the migration of the database tables to be in sync with your models.
makemigrations simply analyzes your current models for any changes that would be out of sync with your database and creates a migrations file that can be used to bring the in sync. If left at this point, your models would still be out of sync with your database possibly breaking your code that queries the database.
migrate is the command to "Make It So!" and apply the changes noted during the makemigrations phase.
Source
This is django's replacement for the old manual south way of making migrations, they can be used to catalog changes in your models and write out changes that will take place in the db.
Migrate is basically the old syncdb but it takes into account all the migrations made by makemigrations.
makemigrations: creates the migrations (generating SQL Command- not yet executed)
migrate: run the migrations (executes the SQL command)
But in your case, Django is asking you to migrate the DEFAULT migrations which should run before first running of server. This would have been the same warning without even creating the first app.
Make migrations : Basically it generate SQL Commands for preinstalled apps and newly created app model which you added in installed app. It dose not executed SQL commands in your database. So actual tables are not created in DB.
Migrate : Migrate execute those SQL commands which are generated by make-migration in Database file . So after migrate all the tables of installed app are created in DB.
According to the second tutorial of the django tutorial series. Migrations are:
The migrate command takes all the migrations that haven’t been applied (Django tracks which ones are applied using a special table in your database called django_migrations) and runs them against your database - essentially, synchronizing the changes you made to your models with the schema in the database.
So pretty much all it does is:
When you execute the make migrations command you're saving the 'instructions' to mysql
When you execute the migrate command, you're executing those same instructions
I want to drop some tables I created for the polls app from the Django tutorials. I am using Django 1.6.5.
I start by turning off the server and then running
manage.py sqlclear
This prints out
BEGIN;
DROP TABLE "polls_choice";
DROP TABLE "polls_poll";
COMMIT;
I then run
manage.py syncdb
When I start the server, go to the admin panel, and click on the models, the values are all still there. Why is this happening? It seems like if I'm dropping those tables, at the very least any existing values should be gone.
According to the documentation, the manage.py sqlclear command only prints queries to execute to clear the database. You need to execute them by yourself.
You can also use manage.py flush to get the empty database (read carefully the documentation, it is actually executing some handlers and adding initial fixtures if you have any)
django-admin.py sqlclear
Prints the DROP TABLE SQL statements for the given app name(s).
The --database option can be used to specify the database for which to print the SQL.
you need to enter here:
python manage.py dbshell
and paste what sqlclear printed out