I've been running a daily dump of a production Django application as follows:
./manage.py dumpdata --exclude=contenttypes --exclude=auth.Permission -e sessions -e admin --all > data.json
Normally, restoring this to another installation for development hasn't caused a problem, but recently attempts to restore the data have caused this:
./manage.py loaddata -i data.json
django.db.utils.IntegrityError: Problem installing fixtures: The row in table 'reversion_version' with primary key '1' has an invalid foreign key: reversion_version.content_type_id contains a value '14' that does not have a corresponding value in django_content_type.id.
This suggests to me that the problem has been caused by the recent addition of django-reversion to the codebase, but I am not sure why and I have not been able to find any means of importing the backup. Some posts suggest that using natural keys may work, but then I get errors like:
django.core.serializers.base.DeserializationError: Problem installing fixture 'data.json': [u"'maintainer' value must be an integer."]
"maintainer" is in this case a reference to this bit of code in a model definition in models.py:
maintainer = models.ForeignKey(Organization,related_name="maintainer",blank=True,null=True)
Does anyone has any suggestions as to how I might get this dump installed, or modify the dump procedure to make a reproducible dump?
I note that the production site is using Postgres and the test site has SQLite, but this has never been a problem before.
On your local machine clone your project and do something like this:
Checkout the project at state that was used to create the dump.
Create a new database and tables.
Load the dump.
Update the code to current state.
Run migrations.
That was rather painful. It seems that the way to fix it was to dump django_content_types as csv from the production posgres database, delete the IDs from the resulting csv file, then do the following on the SQLite database for the test version:
CREATE TABLE temp_table(a, b, c)
.mode csv
.import content_type.csv temp_table
DELETE FROM sqlite_sequence WHERE name = 'django_content_type'
DELETE FROM django_content_type
INSERT INTO django_content_type(name,app_label,model) SELECT * FROM temp_table
That had the effect of setting the ids of the entries in the django_content_type table to match those in the dump, allowing the restore to proceed.
Related
I am running a Django container on docker which depends on Postgres. Now the issue is, when I try to load the postgresql
docker-compose up
cat /tmp/dump.sql | sudo docker exec -i <container_id> psql -U <user> -d <database_name>
This imports the first few tables and gives me a lot of errors on the terminal
An example
constraint "table_id" for relation "table_name" already exists
Another example
null value in column "column_name" violates not-null constraint
Another one
insert or update on table "table_1" violates foreign key constraint "table_id"
I wanted to know is this even the right way to import this data. Plus I cant seem to drop the database as it is already being used by django.
Note:
I tried with volumes where I imported the db with postgresql. But everytime I run django, the database gets reset. I am guessing it has something to do with the migrations, but I can't touch that part.
The issue was being caused because the sql dump was trying to overwrite existing db. A quick solution if you are on dev would be to
docker volume rm <volume_name>.
And then run the cat ... command
I am migrating a project from Django 1.1 to Django 3.0 I am done with the project. When I am dumping the production dump to my local in the newly converted project I get "Table already exists".
Here's what I am doing.
mysql> create database xyx;
docker exec -i <container-hash> mysql -u<user> -p<password> xyx < dbdump.sql
then I run the migrate, as I have had to do some changes to the previously given models.
./manage.py migrate --fake-initial
this is the output I get
_mysql.connection.query(self, query)
django.db.utils.OperationalError: (1050, "Table 'city' already exists")
So, what to do ?
Alright boys and girls, here's the approach I followed to solve this problem.
I dumped the entire database.
docker exec -i <container-hash> mysql -u<username> -p<password> <dbname> < dump.sql
Now I listed all the migrations I made using
./manage.py showmigrations <app-name>
This will give me the list of all the migrations I have applied, now from inspecting the migrations, I realized that from the 7th migration to the 30th migration I had done my changes.
Here's the tedious part which any sys admin can write a script to do in less than 4 lines of bash script. You can generate the raw SQL of any migration with this command.
./manage.py sqlmigrate <app-name> <migration-name> > changes-i-made.sql
Now that I have created my changes-i-made.sql file I'll need to run this script 22 more times but with >> otherwise everytime you run the command with a single > it will keep overwriting your changes file.
Now once all of your migration changes are recorded inside a file, open up your sql shell connect to the database and start pasting the changes or do some sql magic to pick all the changes directly from the file.
Once you're done go ahead and fake all the migrations, cause you don't need Django to do them you already did.
./manage.py migrate --fake
and then login to your production instance and get ready to fuck with your senior team lead who said you couldn't do it.
I just checked to see if this approach is working and the future migrations will be working, so I created one and everything works like a breeze.
My project use an SQLite3 DB but now I need to do a migration from SQLite3 to PostgreSQL. I've used DBeaver for do this using the "Import Data" option. At the end of the importation of all the tables from SQLite3 to PostgreSQL I noticed this error when I try to add a new content:
IntegrityError at /engine/kernel/keyconcept/add/
duplicate key value violates unique constraint
"kernel_keyconcept_pkey" DETAIL: Key (id)=(4) already exists.
If I add a new model, into this project, that use the new DB(PostgreSQL) there aren't problems to add contents, but if I use the old models I see the IntegrityError above.
I've do a test with a model that have only two rows: with the first attempt I received a same error but for key id 1, with the second attempt I received a same error but for key id 2. When I've tried for the third time I was be able to add the content and the same think happens for the next attempts.
So I think that the row counter of the every tables after the migrations never start from the last content but from zero. It strange because previous I've do the same migrations but from SQLite3 to SQLite3 whitout problems.
How I can solve this?
You have to reset the sequences used to generate primary keys. There is a management command that prints the necessary SQL: sqlsequencereset
Note that it only prints out the SQL, you have to execute it manually (which you can probably do with DBeaver).
I've solved using dumpdata and loaddata instead of DBeaver and following this examples all work fine.
In my case I need to exclude some apps as auth and contenttypes, then with dumpdata I've used this:
python3 manage.py dumpdata --exclude contenttypes --exclude auth --indent 2 > db.json
To upload tha datas I've used this:
python3 manage.py loaddata db.json
I wanted to drop a table in my Postgres database on Heroku. But I haven't yet wrapped my head around South, so I decided to write a function:
def dropTable(request):
cursor = connection.cursor()
cursor.execute("DROP TABLE books_request CASCADE;")
success = simplejson.dumps({"success":"success",})
return HttpResponse(success, mimetype="application/json")
So now my books_request table has been dropped.
Then I went into the bash for my app on Heroku, and I did python manage.py syncdb hoping it would recreate the table anew, but it didn't seem to. Why?
Is there any way to force sync just that app?
I also got this error message which might be part of the problem while doing the sync:
Problem installing fixture '/app/bookstore/fixtures/initial_data.json': Traceback (most recent call last):
DatabaseError: Could not load sites.Site(pk=1): value too long for type character varying(50)
UPDATE:
I think the issue is the table isn't dropping in the first place because when I dump the data, I can still see the table. Why isn't it dropping the table properly?
If you're using south you may need to run the migrate command. Additionally if it views the migration as already applied it won't re-run it. Try running the below and then adding the output:
heroku run python manage.py migrate
Yeah you probably didnt drop the table, you just think you have.
Make sure you are referencing connection correctly first by:
from django db import connection
Use django debug toolbar to see the sql that got executed and what happened. Or possibly use a debug program like pudb on the heroku shell to see whats going on.
I have some existing SQL statements that I'd like to use as a custom hook after the CREATE TABLE command, in [myapp]/sql/[model-name].sql.
My app is in INSTALLED_APPS. I see listed if I run ./manage.py sql.
My custom hook is found; I see the SQL statements output if I run any of the following:
./manage.py sqlall <myapp>
./manage.py sqlcustom <myapp>
./manage.py sql <myapp>
I'm using postgres 9.x on my mac.
If I psql to that same database (with no user) and copy them from the .sql file and paste them into the psql command input, they all work... so I believe they're valid SQL understood by postgres. These are all pretty simple INSERT statements (fixtures addressed below).
However, if I run ./manage.py syncdb those statements are either not run, or they are ignored or silent errors happen; all I know is that the new rows do not appear in the database. I am tailing the postgres log file and nothing is logged when I run syncdb, so I don't know if it's not finding my .sql file, or parsing it and finding some error before it gets to the database.
I have created a .json file, for fixtures, with the equivalent of those statements, and ./manage.py loaddata <path-to-json-file> works correctly: my site now shows those values in the database. This makes me believe that my settings file is correct and the database I'm writing to inside postgres is set correctly, and I have write permissions when I run ./manage.py.
I saw in some other post that the django documentation is wrong and I should put the custom hook in the 'models' directory, but I don't know if that's right; if sqlall and sqlcustom find my hook, shouldn't syncdb find it? Also I don't (yet) have a models directory and may not need it.
For various reasons I'd rather not use JSON format, but if I have to I will... however I've invested so much time in the .sql format I really want to know what's going on (and I've seen enough existing related questions that this might help others).
I believe I found it, although it's based on behaviors not any real research. I simply changed 'tile' to 'tilexx' everywhere and it worked. This django-project post indicates that if there is some sort of python class name conflict the custom SQL won't be executed... and 'tile' is a pretty common thing.
So the answer is to change the name of my class to something a bit more unique.
I've been searching for an answer to a similar problem, trying to initialize an sqlite database with data I dumped from a Flask application I'm porting to Django. Like the OP, all of the following did the right thing:
./manage.py sqlall <myapp> ./manage.py sqlcustom <myapp> ./manage.py sql <myapp>
However, the insert statements in myapp/sql/myapp.sql were not being run. A careful reading of the documentation revealed this clue:
Django provides a hook for passing the database arbitrary SQL that’s executed just after the CREATE TABLE statements when you run syncdb.
(emphasis added)
The issue is that I had already registered my models and run syncdb, so the table in question already existed in the database, although it held no data yet. I deduce that because of this, the CREATE TABLE statement was not being run on subsequent executions of syncdb, and therefore, my custom sql could not be run after that statement. The solution was to DROP table table_name and then run syncdb again, at which point my custom sql was run.