I have deployed my project to Heroku and currently trying to load the data dump from local sqlite database to the Heroku database. The remote database is clean and untouched other than the initial migrate command.
I have tried the following combinations of dump but all of them returned an error
python manage.py dumpdata --exclude contenttypes --> data.json
python manage.py dumpdata --exclude auth.permission --exclude contenttypes --indent 2 > data.json
python manage.py dumpdata --exclude auth.permission --exclude contenttypes --exclude auth.user --indent 2 > data.json
and the error is:
django.db.utils.IntegrityError: Problem installing fixture
'/app/data.json': Could not load wellsurfer.Profile(pk=6): duplicate
key value violates unique constraint "wellsurfer_profile_user_id_key"
DETAIL: Key (user_id)=(1) already exists.
i would like to post the json file here but it is about 120,000 lines. But i can provide specific portions if needed. The error clearly says the key exists but the database is clean in the beginning. Obviously, i am doing something very basic thing wrong and i hope you can point me in the right direction. I have tried recommendations that i found on Stackoverflow with no success. How to manage.py loaddata in Django
I had the same problem, and this is what worked for me
source (local sqlite)
python manage.py dumpdata --natural-foreign --indent 4 > datadump.json
(this will include everything, even the auth app / users)
destination (heroku postgres)
python manage.py migrate
python manage.py shell
>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.all().delete()
>>> quit()
Finally, run following command to load the json data:
python manage.py loaddata datadump.json
Related
I have been trying to migrate a database from SQLite to POSTGRESQL.
I am using json fixtures to dump and load the data, I tested multiple ways but I end up in a similar situation, and I say similar because I can reach 2 slightly different errors.
So the 2 errors I can reach are the following:
django.db.utils.IntegrityError: Problem installing fixture '/PATH/wcontenttypes.json': Could not load MyApp.DIN_STATUS(pk=1): duplicate key value violates unique constraint "MyApp_din_status_DSP_id_EA_id_1c1c3e96_uniq"
DETAIL: Key ("DSP_id", "EA_id")=(542, 20324) already exists.
The other one is the same but instead of pk=1, its pk=5
What did I check?
If there's a duplicate -> but there is not
If the row referenced by the id exist -> it does exist
Removing the row that gives the error -> The next one gives the error (in case its pk=5, then pk=6, if pk=1 then pk=2)
What did I test?
I did multiple test looking around the internet, and testing almost anything I could find, the research ended up with 3 main ideas on how to do this
Test 1
python manage.py dumpdata > wcontenttypes.json
#-Swap to postgre
find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc" -delete
python manage.py makemigrations
python manage.py migrate
python manage.py shell
>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.all().delete()
python manage.py loaddata wcontenttypes.json -v3
Test 2
python manage.py dumpdata > wcontenttypes.json
#-Swap to postgre
python manage.py migrate --run-syncdb
python manage.py shell
>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.all().delete()
python manage.py loaddata wcontenttypes.json -v3
Test 3 (I played a lot more than what is shown in this example, like only excluding content types or only having natural-foreign and removing contenttypes from the shell)
python manage.py dumpdata --natural-foreign --exclude contenttypes --exclude auth.permission --exclude admin.logentry --exclude sessions.session --indent 4 > fixture.json
#-Swap to postgre
find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc" -delete
python manage.py makemigrations
python manage.py migrate
python manage.py loaddata fixture.json -v3
Right now I am pretty much lost as everything I see in the internet is something I already tested.
If you miss any info let me know and I will make an EDIT.
So reading more posts I got to this one:
https://gist.github.com/sirodoht/f598d14e9644e2d3909629a41e3522ad
After reading some of the comments I remembered that the dev team got signals creating this din_status post_save, so they are the reason of duplicate entry.
Commenting the signals involved fixed the issue.
Here is how I tried to dump mysql DB:
python3 manage.py dumpdata > dumpdata.json
Then, I tried to reload it:
python3 manage.py loaddata dumpdata.json
This is the error that I get:
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 226398200 (char 226398199)
django.core.serializers.base.DeserializationError: Problem installing fixture '/home/dumpdata.json':
The usual case is when you print something during your application boostrap.
Those things you print will eventually end up in your fixture.json, messing up your JSON Structure.
To make things more readable, you can use:
python3 manage.py dumpdata --indent 4 --natural-primary --natural-foreign -e contenttypes -e auth.Permission -e sessions > dumpdata.json
Some details:
--indent 4 will "pretify" your JSON output, it'll be way easier to checkup your Data
-e sessions will remove session's app data. Same for contenttype which often mess things up, admin and auth.Permission
--natural-primary --natural-foreign will use some more natural PKs instead of IDs (eg when you have some unique=True fields)
how to copy records from one database to another django ?
I tried for the first database
python manage.py dumpdata material - indent = 1
material is the directory database
after ?
material.json ?
do I copy this file somewhere? in the second database ?
You can use this command to dump the data to a json file:
python manage.py dumpdata material --indent=1 > my_dir/material.json
And then this command to load it into the database:
python manage.py loaddata my_dir/material.json
How to get all my data from model to JSON? I need export and import my data
Is there any command?
You can use dumpdata command to dump data in your table. By default it gives in JSON format.
You can do in command line, to print data on screen.
$ python manage.py dumpdata
As suggested by Rohan, you need the dumpdata command.
I generally do an app at a time, output to file, and add an indent to the output to make it more readable -
$ python manage.py dumpdata --indent 2 myapp > /path/to/myapp/fixtures/my_data.json
First create fixtures folder inside your app.
Dump all models data of the app:
python manage.py dumpdata --format=json --indent=4 [app_name] > [app_name]/fixtures/initial_data.json
In Django-South:
I changed I've run the initial migration successfully for myapp but for some reason, after I've made a change to my model and go to
./manage.py schemamigration myapp --auto
./manage.py migrate myapp
And I get a lot of traceback which ends in:
(1050, "Table 'my_table' already exists")
After much googling, I found and tried this:
./manage.py migrate myapp --fake
And then I proceed to migrate it, to no avail; same error.
Any suggestions?
I just got this same error, and found this question by search.
My problem was that my second migration I'd created using the --initial flag, i.e.
$ ./manage.py startapp foo
$ ./manage.py schemamigration --initial foo
$ ./manage.py migrate foo
... make some changes to foo ...
$ ./manage.py schemamigration --initial foo
(oops!)
$ ./manage.py migrate foo
... and I get the error, and the migration fails because in the second migration, South is trying to create a table its already created.
Solution
In my migrations folder:
$ ls foo/migrations
0001_initial.py 0002_initial.py
remove that second migration and re-export the second migration with the correct --auto flag:
$ rm foo/migrations/0002_initial.py
$ ./manage.py schemamigration --auto foo
$ ./manage.py migrate foo
Success!
There may be other things that cause this error, but that was my bad!
Is it an existing app?
In that case you will need to convert it in addition to the fake bit.
There are good docs here on converting an existing app.
Although they are quite tricky to find if you don't know where they are already ( ;
For converting, after adding south to your installed apps:
./manage.py syncdb
./manage.py convert_to_south myapp
./manage.py migrate myapp 0001 --fake
this problem actually happens if one of the cases:
1) You made "schemamigration app_name --initial" after one is "--auto"
2) You interrupted the last migration you have made.
To resolve such problem you apply the following:
1) mark your last schema migration as fake.
python manage.py schemamigration app_name --fake
Note: Make sure that the schema of models is same as schema of tables in database.
2) apply the migration again by doing
python manage.py schemamigration app_Name --auto
python manage.py migrate app-Name
Note: sometimes you might add manually a specific field you already added using the following syntax.
python manage.py schemamigration app_name --add-field My_model.added_field
For more info. regarding south, you could check its documentation here.