I've being fighting with this command for several hours now.
If I do
python manage.py dumpdata --natural-foreign --> data.json
when I loaddata I get the error
Could not load contenttypes.ContentType(pk=19): duplicate key value
violates unique constraint
"django_content_type_app_label_76bd3d3b_uniq" DETAIL: Key (app_label,
model)=(misuper, stockitem) already exists.
Then if I do
python manage.py dumpdata --natural-foreign --exclude=contenttypes --> data.json
I get a similar error but with a ̀£auth.Permission object:
Could not load auth.Permission(pk=55): duplicate key value violates
unique constraint "auth_permission_content_type_id_01ab375a_uniq"
And if I do
python manage.py dumpdata --natural-foreign --exclude=contenttypes --exclude=auth --> data.json
when I loaddata I get
User matching query does not exist
Of course, I'm excluding the auth table.
So ... WTF can I do to load the data? All my tests depend on this.
I believe the docs are insufficient. I'm stuck here, please help.
Try it like this:
python manage.py dumpdata --natural-foreign \
--exclude=auth.permission --exclude=contenttypes \
--indent=4 > data.json
Watch out for #receiver in your models. They might be your issue, as they were mine.
I just want to point out my case here. I had a receiver in my model.
Basically, to create a few instances for extra data to my user, like so.
#receiver(post_save, sender=User)
def create_user_profile(sender, instance, created, **kwargs):
if created:
Profile.objects.create(account=instance)
Preferences.objects.create(account=instance)
Running loaddata breaks into "duplicate key value" because my fixture have these rows and my receiver tries to do it as well. Commenting those lines out, running the command and uncommenting did the trick for me.
Maybe there is some sort of flag for "disable receivers" that I do not know about. This seems like something too expected to not be solved by the Django crew.
Related
I currently have a database using Heroku and want to migrate it to AWS, where both use PostgresSQL. So, after some digging on how to get it done I followed the steps as on this youtube video.
I initially ran python manage.py dumpdata > dumpdata.json with my Heroku database credentials in Django.
Afterwards, I changed my database credentials in settings.py to the AWS database credentials, and ran python manage.py migrate --run-syncdb which worked successfully.
And then I ran the code python manage.py loaddata dumpdata.json, when where I was thrown an error.
The following error came up:
django.db.utils.IntegrityError: Problem installing fixture 'C:\Users\Acer\Desktop\Web Development\Proffesional\eblossom\eblossom\eblossom\dumpdata.json': Could not load contenttypes.ContentType(pk=9): duplicate key value violates unique constraint "django_content_type_app_label_model_76bd3d3b_uniq"
DETAIL: Key (app_label, model)=(user, profile) already exists.
I don't understand what has gone wrong over here, the site is working perfectly fine all this time with no database compilation error, but now when I try to migrate it to AWS, I am thrown with this problem.
Just in case my models.py:
class Profile (models.Model):
user = models.OneToOneField(User, on_delete=models.CASCADE)
mobile_number = models.CharField(max_length=12, null=True)
guest = models.BooleanField(default=False)
def __str__(self):
return f"{self.user}"
I do this all the time between databases, local and remote, postgres or sqlite.
In order to achieve that, do the following steps in order :
# Start the same as you did on local database
python manage.py dumpdata > db.json
# push this file to your production/server location
# and delete or start with a fresh new database
python manage.py migrate
python manage.py shell
# Enter the following in the shell
from django.contrib.contenttypes.models import ContentType
ContentType.objects.all().delete()
# Exit shell and run following command
python manage.py loaddata db.json
I want to load a fixture for my selenium testing. Using fixtures was successful in my initial tests, so I know I am capable of loading the fixtures in my test setup and using them in my tests. I have attempted several approaches.
First, I generated fixtures specific to the models I was testing using dumpdata. An example is below:
python manage.py dumpdata protocols.Step --indent=2 > functional_tests/fixtures/step.json
When used in my test as so:
class SignInTest(FunctionalTest):
fixtures = ['admin_user.json', 'protocol.json', 'step.json',
'protocol_element.json']
def test_login_and_view_user_data(self):
...
I get error:
django.db.utils.IntegrityError: Problem installing fixtures: The row in table 'protocols_protocolelement' with primary key '37' has an invalid foreign key: protocols_protocolelement.element_content_type_id contains a value '41' that does not have a corresponding value in django_content_type.id.
Second attempt involved using all the test data in my tables, but excluding contenttypes:
python manage.py dumpdata --indent=2 -e contenttypes > functional_tests/fixtures/initial_data.json
class SignInTest(FunctionalTest):
fixtures = ['initial_data.json']
...
Getting the error:
django.db.utils.OperationalError: Problem installing fixture '.../mike/mike/functional_tests/fixtures/initial_data.json': Could not load auth.Permission(pk=103): no such table: auth_permission
Next, I tried using natural to show the natural keys:
python manage.py dumpdata --natural -e contenttypes -e auth.Permission --indent=2 > functional_tests/fixtures/initial_data2.json
Only to get the error:
django.db.utils.OperationalError: Problem installing fixture '.../mike/mike/functional_tests/fixtures/initial_data.json': Could not load auth.User(pk=1): no such table: auth_user
Noticing natural was depreciated I tried --natural-foreign and wanted to include user and permission models (I need contenttypes for my models anyway):
python manage.py dumpdata --natural-foreign --indent=2 > functional_tests/fixtures/initial_data3.json
Only to get the error:
django.db.utils.IntegrityError: Problem installing fixture '.../mike/mike/functional_tests/fixtures/initial_data3.json': Could not load contenttypes.ContentType(pk=35): UNIQUE constraint failed: django_content_type.app_label, django_content_type.model
So, any ideas on how to load the fixture so I can run my tests? Is there something simple I'm missing? Thanks!
After some more reading about how Django maintains its own models and such, it is my understanding that Django caches the contenttype, auth.Permission, etc and uses them in testing frameworks (I was using StaticLiveServerTestCase). This means that when I was loading my fixture, it was clashing with the data Django had stored for its own uses causing the integrity error. This is what worked for me:
python manage.py dumpdata -e contenttypes -e admin -e auth.Permission --natural-foreign --indent=2 > functional_tests/fixtures/initial_data4.json
This post has some additional helpful information to help me solve the problem:
Problems with contenttypes when loading a fixture in Django.
All this exercises around dumpdata didn't bring me any result.
But I found this advise:
https://stackoverflow.com/a/3855764/12191031
For some reason reset doesn't work in my code, but it was enough load fixture before every test and flush DB after each one.
So, my solution is this:
from django.core import management
from django.contrib.staticfiles.testing import StaticLiveServerTestCase
from selenium import webdriver
class login_page_test(StaticLiveServerTestCase):
#classmethod
def setUpClass(cls):
cls.selenium = webdriver.Firefox()
cls.selenium.implicitly_wait(5)
super(login_page_test, cls).setUpClass()
#classmethod
def tearDownClass(cls):
cls.selenium.quit()
super(login_page_test, cls).tearDownClass()
def setUp(self):
# for every test I load fixture
management.call_command("loaddata", "user-data.yaml", verbosity=0)
super(login_page_test, self).setUp()
def tearDown(self):
# after each test I flush all DB
management.call_command("flush", verbosity=0, interactive=False)
super(login_page_test, self).setUp()
I'm happy )))
How do I delete all the data in the database for on Django app? In previous version manage.py reset APPNAME did the job, but that's been deprecated.
What are we supposed to do now if we want to delete all the data from an app using the command line?
reset and sqlreset were both just wrappers around other management commands. sqlreset in particular can be duplicate by simply running:
python manage.py sqlclear myapp
python manage.py sqlall myapp
reset only served to automatically run the result of sqlreset on the database. Personally, I think removing that is a fantastic idea. Still, if you want similar functionality, you can just pipe the output to your database's shell commands.
For PostgreSQL, for example:
python manage.py sqlclear myapp | psql mydatabase
python manage.py sqlall myapp | psql mydatabase
If you want single command that should work with most database types you can pipe the drop table statements, that sqlclear generates, to dbshell
python manage.py sqlclear myapp | python manage.py dbshell
from django.contrib.contenttypes.models import ContentType
for ct in ContentType.objects.all()
ct.model_class().objects.all().delete()
Now that Django integrates migrations by default, you first need to make migrations for your app are first unapplied then deleted.
Here is the command line that works at least with Django 1.8 (replacing by the application you want to delete all associated data and:
# First, update the DB so it thinks no migrations were applied to the app
python manage.py migrate --fake <app_name> zero
# Erase all migrations in the app folder
rm -r "<app_name>/migrations/*"
# Erase the application tables
python manage.py sqlclear <app_name> | python manage.py dbshell
# Recreate the app tables, that will be empty
python manage.py makemigrations <app_name>
python manage.py migrate <app_name>
DIY
If you want to do that from the command line, create the following custom command:
from django.core.management.base import AppCommand, CommandError
from django.utils.six.moves import input
from django.db import DEFAULT_DB_ALIAS, connections
class Command(AppCommand):
help = (
'Removes ALL DATA related to the given app from the database '
'by calling model.objects.all().delete() for all app models. '
'This also removes related data in other apps via cascade.'
)
def add_arguments(self, parser):
super(Command, self).add_arguments(parser)
parser.add_argument(
'--noinput', '--no-input',
action='store_false', dest='interactive', default=True,
help='Tells Django to NOT prompt the user for input of any kind.',
)
parser.add_argument(
'--database', action='store', dest='database', default=DEFAULT_DB_ALIAS,
help='Nominates a database to reset. Defaults to the "default" database.',
)
def handle_app_config(self, app_config, **options):
app_label = app_config.label
database = options['database']
interactive = options['interactive']
db_name = connections[database].settings_dict['NAME']
confirm = (ask_confirmation(app_label, db_name)
if interactive else 'yes')
if confirm == 'yes':
for model in app_config.get_models():
model.objects.using(database).all().delete()
self.stdout.write('Reset done.\n')
else:
self.stdout.write("Reset cancelled.\n")
def ask_confirmation(app_label, db_name):
return input("""You have requested a reset of the application {app_label}.
This will IRREVERSIBLY DESTROY all data related to the app currently in
the {db_name} database, and return each table to empty state.
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: """.format(**locals()))
Copy it to app/management/commands folder in any of your apps folders and run it with
./manage.py app_db_tables_reset any_installed_app_name
Ready-made package
The command is available in the django_commands package, you can install it with
pip install git+http://github.com/mrts/django-commands.git
and add it to INSTALLED_APPS to activate the command.
Tested with Django 1.9, it may or may not work with 1.8.
I am unable to use syncdb because my app uses some MySQL views. I have run manage.py sqlall <app>, but this does not output the SQL for django_content_type table or the auth_permission tables. I have also had a look into south and django evolution, but they both require syncdb, and I'm not sure they would help anyway.
I have manually added some models to the tables, but this is getting frustrating, and having installed the dbsettings app I am unsure of what I now need to enter.
Does anyone know of a way to get manage.py (or something else) to output the SQL for these tables and their contents?
Thanks.
Having done a bit more digging, I found these:
Fixing the auth_permission table after renaming a model in Django and manage.py sql command for django models - Django.
These output the tables, but not the data:
python manage.py sql auth
python manage.py sql admin
But this gets a lot closer. In the end I managed it with the following:
from django.contrib.auth.management import create_permissions
from django.db.models import get_apps
for app in get_apps():
create_permissions(app, None, 2)
from django.contrib.contenttypes.management import update_all_contenttypes
update_all_contenttypes(interactive=True)
This adds all the permissions and then all the content types which are needed. interactive=True means that it asks you if you want to remove stale content types.
#hajamie solution works for older supported version, taking a hint, below is what worked for me!
django = 1.9.7
from django.contrib.auth.management import create_permissions
from django.contrib.auth.models import Permission
from django.apps import apps
def fix_user_permission():
"""
run this method via shell whenever any amendments in any of the tables is made
"""
print "fixing user permissions"
# delete pre-existing user permission
Permission.objects.all().delete()
apps.models_module = True
create_permissions(apps, verbosity=0)
apps.models_module = None
print "process completed - fixed user permissions"
The easiest solution I found is to install Django Extensions, add it to settings.INSTALLED_APPS and run:
manage.py update_permissions
With what command can I delete tables from a django db of a specific app/ only one table?
I did find all sorts of things, but not how to get rid of a table.
Also while I am at this.
I create the models and hit syncdb and then I have the tables.
If I want to update/add to those tables, do I run into problems?
Your best bet would be to get django-south installed in your machine.
if you are using pip, do pip install django-south
This allows you too migrate data forward and backwards.
This is very handy especially if you need to update tables, and new tables etc.
check it out.
adding south to apps are as easy as python manage.py schemamigration appname --initial
Make your changes in a model and run the following python manage.py schemamigration appname --auto
Once your data migration file has been created it'll tell you data is now ready to migrate.
Simply use python manage.py migrate appname
http://south.aeracode.org/docs/about.html
Hope this helps
If you are deleting a table, this is done in the model file of the specific app that you are trying to delete, there is no command for this, you just go into the file and delete it and then re-run syncdb, for your other question, the answer is the same.. every app folder should have a file called "models.py" and here is where the models which are, in this case, equivalent to tables are specified, along with their fields, you simply edit this to make any changes.
def reset():
import install
from django.db.models import get_models
removed_tables = []
exceptions = []
for model in get_models():
if model.__name__ not in ('User','Session','Group','Permission'):
try:
model.objects.all().delete() # So we can remove the table without complaints from the database server.
CURSOR.execute('DROP TABLE %s' % model._meta.db_table)
print "Dropped table %s from model %s" % (model._meta.db_table, model.__name__)
except Exception as e:
exceptions.append([model._meta.db_table, str(e)])
continue
removed_tables.append(model._meta.db_table)
print "Removed %s tables" % len(removed_tables)
syncdb()
install.install() # A function that leads to the creation of my default data