I have been trying to migrate a database from SQLite to POSTGRESQL.
I am using json fixtures to dump and load the data, I tested multiple ways but I end up in a similar situation, and I say similar because I can reach 2 slightly different errors.
So the 2 errors I can reach are the following:
django.db.utils.IntegrityError: Problem installing fixture '/PATH/wcontenttypes.json': Could not load MyApp.DIN_STATUS(pk=1): duplicate key value violates unique constraint "MyApp_din_status_DSP_id_EA_id_1c1c3e96_uniq"
DETAIL: Key ("DSP_id", "EA_id")=(542, 20324) already exists.
The other one is the same but instead of pk=1, its pk=5
What did I check?
If there's a duplicate -> but there is not
If the row referenced by the id exist -> it does exist
Removing the row that gives the error -> The next one gives the error (in case its pk=5, then pk=6, if pk=1 then pk=2)
What did I test?
I did multiple test looking around the internet, and testing almost anything I could find, the research ended up with 3 main ideas on how to do this
Test 1
python manage.py dumpdata > wcontenttypes.json
#-Swap to postgre
find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc" -delete
python manage.py makemigrations
python manage.py migrate
python manage.py shell
>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.all().delete()
python manage.py loaddata wcontenttypes.json -v3
Test 2
python manage.py dumpdata > wcontenttypes.json
#-Swap to postgre
python manage.py migrate --run-syncdb
python manage.py shell
>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.all().delete()
python manage.py loaddata wcontenttypes.json -v3
Test 3 (I played a lot more than what is shown in this example, like only excluding content types or only having natural-foreign and removing contenttypes from the shell)
python manage.py dumpdata --natural-foreign --exclude contenttypes --exclude auth.permission --exclude admin.logentry --exclude sessions.session --indent 4 > fixture.json
#-Swap to postgre
find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc" -delete
python manage.py makemigrations
python manage.py migrate
python manage.py loaddata fixture.json -v3
Right now I am pretty much lost as everything I see in the internet is something I already tested.
If you miss any info let me know and I will make an EDIT.
So reading more posts I got to this one:
https://gist.github.com/sirodoht/f598d14e9644e2d3909629a41e3522ad
After reading some of the comments I remembered that the dev team got signals creating this din_status post_save, so they are the reason of duplicate entry.
Commenting the signals involved fixed the issue.
I have deployed my project to Heroku and currently trying to load the data dump from local sqlite database to the Heroku database. The remote database is clean and untouched other than the initial migrate command.
I have tried the following combinations of dump but all of them returned an error
python manage.py dumpdata --exclude contenttypes --> data.json
python manage.py dumpdata --exclude auth.permission --exclude contenttypes --indent 2 > data.json
python manage.py dumpdata --exclude auth.permission --exclude contenttypes --exclude auth.user --indent 2 > data.json
and the error is:
django.db.utils.IntegrityError: Problem installing fixture
'/app/data.json': Could not load wellsurfer.Profile(pk=6): duplicate
key value violates unique constraint "wellsurfer_profile_user_id_key"
DETAIL: Key (user_id)=(1) already exists.
i would like to post the json file here but it is about 120,000 lines. But i can provide specific portions if needed. The error clearly says the key exists but the database is clean in the beginning. Obviously, i am doing something very basic thing wrong and i hope you can point me in the right direction. I have tried recommendations that i found on Stackoverflow with no success. How to manage.py loaddata in Django
I had the same problem, and this is what worked for me
source (local sqlite)
python manage.py dumpdata --natural-foreign --indent 4 > datadump.json
(this will include everything, even the auth app / users)
destination (heroku postgres)
python manage.py migrate
python manage.py shell
>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.all().delete()
>>> quit()
Finally, run following command to load the json data:
python manage.py loaddata datadump.json
how to copy records from one database to another django ?
I tried for the first database
python manage.py dumpdata material - indent = 1
material is the directory database
after ?
material.json ?
do I copy this file somewhere? in the second database ?
You can use this command to dump the data to a json file:
python manage.py dumpdata material --indent=1 > my_dir/material.json
And then this command to load it into the database:
python manage.py loaddata my_dir/material.json
How to get all my data from model to JSON? I need export and import my data
Is there any command?
You can use dumpdata command to dump data in your table. By default it gives in JSON format.
You can do in command line, to print data on screen.
$ python manage.py dumpdata
As suggested by Rohan, you need the dumpdata command.
I generally do an app at a time, output to file, and add an indent to the output to make it more readable -
$ python manage.py dumpdata --indent 2 myapp > /path/to/myapp/fixtures/my_data.json
First create fixtures folder inside your app.
Dump all models data of the app:
python manage.py dumpdata --format=json --indent=4 [app_name] > [app_name]/fixtures/initial_data.json
Is there anyway to load one fixture and have it load multiple fixtures?
I'd ideally like to type:
python manage.py loaddata all_fixtures
And have that load all of the data instead of having to type everything. Is this possible?
Using $ python manage.py loaddata myfixtures/*.json would work as Bash will substitute the wildcard to a list of matching filenames.
I have multiple apps on the project directory and have each app with its 'fixtures' directory. So using some bash I can do:
python3 manage.py loaddata */fixtures/*.json
And that expands all of the json files inside of the fixtures directory on each app in my project. You can test it by simply doing:
ls */fixtures/*.json
Why not create a Makefile that pulls in all your fixtures? eg something like:
load_all_fixtures:
./manage.py loaddata path/to/fixtures/foo.json
./manage.py loaddata path/to/fixtures/bar.json
./manage.py loaddata path/to/fixtures/baz.json
And then at the shell prompt, run
make load_all_fixtures
(This kind of approach is also good for executing unit tests for certain apps only and ignoring others, if need be)
This thread shows up among the first results with a Google search "load data from all fixtures" and doesn't mention what IMO is the correct solution for this, ie the solution that allows you to load any fixtures you want without any wildcard tricks nor a single modification of the settings.py file (I also used to do it this way)
Just make your apps' fixtures directories flat (and not the usual Django scheme that e.g. goes app_name/templates/app_name/mytemplate.html), ie app_name/fixtures/myfixture.[json, yaml, xml]
Here's what the django doc says :
For example:
django-admin loaddata foo/bar/mydata.json
would search /fixtures/foo/bar/mydata.json for each installed application, /foo/bar/mydata.json for each directory in FIXTURE_DIRS, and the literal path foo/bar/mydata.json.
What that means is that if you have a fixtures/myfixture.json in all your app directories, you just have to run
./manage.py loaddata myfixture
to load all the fixtures that are located there within your project ... And that's it ! You can even restrict what apps you load fixtures from by using --app or --exclude arguments.
I'll mention that I use my fixtures only to populate my database while doing some development so I don't mind having a flat structure in my 'fixtures' directories ... But even if you use your fixtures for tests it seems like having a flat structure is the Django-esque way to go, and as
that answer suggests, you would reference the fixture from a specific app by just writing something like :
class MyTestCase(TestCase):
fixtures = ['app_name/fixtures/myfixture.json']
My command is this, simple. (django 1.6)
python manage.py loaddata a.json b.json c.json
If you want to have this work on linux and windows you simply could use this for loading all your json-Fixtures:
import os
files = os.listdir('path/to/my/fixtures')
def loaddata(file):
if os.path.splitext(file)[1] == '.json' and file != 'initial_data.json':
print file
os.system("python manage.py loaddata %s" % file)
map(loaddata, files)
Works for me with Django-admin version 3.1.4
python manage.py loaddata json_file_1 json_file_2
My folder structure is like this -
app_name_1
├──fixtures
├────json_file_1.json
├────json_file_2.json
app_name_2
├──fixtures
├────json_file_3.json
Manage.py loaddata will look automatically in certain places, so if you name your fixtures the same in each app, or put all your fixtures in the same folder it can be easy to load them all. If you have many different fixtures, and need a more complex naming schema, you can easily load all your fixtures using find with -exec
find . -name "*.json" -exec manage.py loaddata {} \;
As I state in this [question][2], I also have this in a fabfile. EDIT: use python manage.py if manage.py is not in your VE path.
If your fixtures are located into the same folder, you can simply ls and xargs: ls myfolder | xargs django-admin loaddata.
Example with this structure:
$ tree fixtures/
root_dir/fixtures/
├── 1_users.json
├── 2_articles.json
└── 3_addresses.json
$ ls -d fixtures/* | xargs django-admin loaddata
would do the same as:
$ django-admin loaddata 1_users.json
$ django-admin loaddata 2_articles.json
$ django-admin loaddata 3_addresses.json
After doing a bit of searching, I ended up writing this script. It searches through all directories named "fixtures" for .json files and runs a "python manage.py loaddata {fixture_name}.json". Sometimes ordering matters for foreign key constraints, so it leaves a fixture in the queue if the constraint cannot be resolved.
(Note: It requires the pip package simple_terminal that I wrote. And I set it up to be run by 'python manage.py runscript ', which requires django-extensions.)
# load_fixture.py
#
# A script that searches for all .json files in fixtures directories
# and loads them into the working database. This is meant to be run after
# dropping and recreating a database then running migrations.
#
# Usage: python manage.py runscript load_fixtures
from simple_terminal import Terminal
from django.core.management import call_command
from django.db.utils import IntegrityError
def load_fixture(fixture_location):
# runs command: python manage.py loaddata <fixture_location>
call_command('loaddata', fixture_location)
def run():
with Terminal() as t:
# get all .json files in a fixtures directory
fixture_locations = t.command(
'find . -name *.json | grep fixtures | grep -v env')
while fixture_locations:
# check that every iteration imports are occuring
errors = []
length_before = len(fixture_locations)
for fl in fixture_locations:
try:
# try to load fixture and if loaded remove it from the array
load_fixture(fl)
print("LOADED: {}".format(fl))
fixture_locations.remove(fl)
except IntegrityError as e:
errors.append(str(e))
# if import did not occur this iteration raise exception due to
# missing foreign key reference
length_after = len(fixture_locations)
if length_before == length_after:
raise IntegrityError(' '.join(errors))
This works well for me; it finds all files located in fixtures directories within the the src directory:
python manage.py loaddata \
$(ls -1 src/**/fixtures/* | tr '\n' '\0' | xargs -0 -n 1 basename | tr '\n' ' ')
python manage.py loaddata ./*/fixtures/*.json
This command will look for the folder fixture in all the directory and then it will pick up all the files with json extension and will install the fixtures.
This way you won't have to have just one folder for fixtures instead you can have fixtures on app level and in multiple apps.
It will work with the ideal folder structure like this -
app_name_1
├──fixtures
├────json_file_1.json
├────json_file_2.json
app_name_2
├──fixtures
├────json_file_3.json