Related
I currently have a database using Heroku and want to migrate it to AWS, where both use PostgresSQL. So, after some digging on how to get it done I followed the steps as on this youtube video.
I initially ran python manage.py dumpdata > dumpdata.json with my Heroku database credentials in Django.
Afterwards, I changed my database credentials in settings.py to the AWS database credentials, and ran python manage.py migrate --run-syncdb which worked successfully.
And then I ran the code python manage.py loaddata dumpdata.json, when where I was thrown an error.
The following error came up:
django.db.utils.IntegrityError: Problem installing fixture 'C:\Users\Acer\Desktop\Web Development\Proffesional\eblossom\eblossom\eblossom\dumpdata.json': Could not load contenttypes.ContentType(pk=9): duplicate key value violates unique constraint "django_content_type_app_label_model_76bd3d3b_uniq"
DETAIL: Key (app_label, model)=(user, profile) already exists.
I don't understand what has gone wrong over here, the site is working perfectly fine all this time with no database compilation error, but now when I try to migrate it to AWS, I am thrown with this problem.
Just in case my models.py:
class Profile (models.Model):
user = models.OneToOneField(User, on_delete=models.CASCADE)
mobile_number = models.CharField(max_length=12, null=True)
guest = models.BooleanField(default=False)
def __str__(self):
return f"{self.user}"
I do this all the time between databases, local and remote, postgres or sqlite.
In order to achieve that, do the following steps in order :
# Start the same as you did on local database
python manage.py dumpdata > db.json
# push this file to your production/server location
# and delete or start with a fresh new database
python manage.py migrate
python manage.py shell
# Enter the following in the shell
from django.contrib.contenttypes.models import ContentType
ContentType.objects.all().delete()
# Exit shell and run following command
python manage.py loaddata db.json
I have been trying to synchronize changes to a model in a Django app using migrations in 1.7 (postgres 9.1 - let me know if you need more details of my environment), but manage.py migrate doesn't seem to do anything, and sqlmigrate doesn't emit any SQL.
I thought Django 1.7 - "No migrations to apply" when run migrate after makemigrations might be applicable to my situation, and I did find some history in the django_migrations table in my database. I deleted the records for the app I am trying to migrate.
Recently I gave up on getting the alter table statements to generate/run and dropped the original version of the table. And while manage.py migrate states it is applying the migration, nothing happens to the database.
Here are the steps I've been trying:
Delete the history.
rm -r myapp/migrations
../manage.py dbshell
myapp_db=> delete from django_migrations where app='myapp'
Create an initial migration.
cp myapp/models.py.orig myapp/models.py
../manage.py makemigrations myapp
../manage.py migrate
manage.py migrate returns the following:
....
Running migrations:
Applying myapp.0001_initial... FAKED
Then I swap in the new models and generate a new migration.
cp myapp/models.py.new myapp/models.py
../manage.py makemigrations myapp
The result of makemigrations is in myapp/migrations/0002_notificationlog.py:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import models, migrations
class Migration(migrations.Migration):
dependencies = [
('myapp', '0001_initial'),
]
operations = [
migrations.CreateModel(
name='NotificationLog',
fields=[
('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
('tstamp', models.DateTimeField(help_text=b'Log time', auto_now_add=True)),
('recipient', models.CharField(max_length=100)),
('subject', models.TextField()),
],
options={
},
bases=(models.Model,),
),
]
Run this migration:
../manage.py migrate
manage.py migrate acts like everything is OK:
....
Running migrations:
Applying myapp.0002_notificationlog... OK
I can see the log entries appear in django_migrations, but the table is not created.
I'm lost. Any idea what to try next?
Update
When running migrate -v 3 as requested, I see
Running pre-migrate handlers for application auth
followed by a similar line for each installed app.
Then
Loading 'initial_data' fixtures...
Checking '/var/www/environment/default/myproj/myproj' for fixtures...
No fixture 'initial_data' in '/var/www/environment/default/myproj/myproj'.
repeated a total of 13 times, the number of unmanaged apps.
Then
Running migrations:
Applying myapp.0001_initial... FAKED
followed by
Running post-migrate handlers for application auth
with a similar line for each installed app.
For migration 0002, the output is the same, except for
Running migrations:
Applying myapp.0002_notificationlog... OK
Note also that sqlmigrate doesn't output anything either:
../manage.py sqlmigrate myapp 0002 -v 3
Produces nothing at all.
Update 2
I copied myapp into a new project and was able to run migrations on it, but migrations stopped working when I imported my main project settings. Are there settings I should be aware of that could affect migration execution, particularly if I've been using South with previous versions of Django?
The problem disappeared with generic project settings and reappeared with my old, complex project settings. I tracked the problem down to a database Router class that was missing an allow_migrate method.
DATABASE_ROUTERS = [ 'myproj.routers.DatabaseAppsRouter', ]
I use this router to handle queries for a separate app in the project (readonly/MySQL).
Sadly I can't blame anyone but myself, since the Django documentation states clearly:
Note that migrations will just silently not perform any operations on a model for which [allow_migrate] returns False. (link)
I had created this router some time ago and didn't add the allow_migrate method to my router class when I upgraded to Django 1.7. When I added the method and made sure it returned True when needed, migrations run and the problem is solved.
How do I delete all the data in the database for on Django app? In previous version manage.py reset APPNAME did the job, but that's been deprecated.
What are we supposed to do now if we want to delete all the data from an app using the command line?
reset and sqlreset were both just wrappers around other management commands. sqlreset in particular can be duplicate by simply running:
python manage.py sqlclear myapp
python manage.py sqlall myapp
reset only served to automatically run the result of sqlreset on the database. Personally, I think removing that is a fantastic idea. Still, if you want similar functionality, you can just pipe the output to your database's shell commands.
For PostgreSQL, for example:
python manage.py sqlclear myapp | psql mydatabase
python manage.py sqlall myapp | psql mydatabase
If you want single command that should work with most database types you can pipe the drop table statements, that sqlclear generates, to dbshell
python manage.py sqlclear myapp | python manage.py dbshell
from django.contrib.contenttypes.models import ContentType
for ct in ContentType.objects.all()
ct.model_class().objects.all().delete()
Now that Django integrates migrations by default, you first need to make migrations for your app are first unapplied then deleted.
Here is the command line that works at least with Django 1.8 (replacing by the application you want to delete all associated data and:
# First, update the DB so it thinks no migrations were applied to the app
python manage.py migrate --fake <app_name> zero
# Erase all migrations in the app folder
rm -r "<app_name>/migrations/*"
# Erase the application tables
python manage.py sqlclear <app_name> | python manage.py dbshell
# Recreate the app tables, that will be empty
python manage.py makemigrations <app_name>
python manage.py migrate <app_name>
DIY
If you want to do that from the command line, create the following custom command:
from django.core.management.base import AppCommand, CommandError
from django.utils.six.moves import input
from django.db import DEFAULT_DB_ALIAS, connections
class Command(AppCommand):
help = (
'Removes ALL DATA related to the given app from the database '
'by calling model.objects.all().delete() for all app models. '
'This also removes related data in other apps via cascade.'
)
def add_arguments(self, parser):
super(Command, self).add_arguments(parser)
parser.add_argument(
'--noinput', '--no-input',
action='store_false', dest='interactive', default=True,
help='Tells Django to NOT prompt the user for input of any kind.',
)
parser.add_argument(
'--database', action='store', dest='database', default=DEFAULT_DB_ALIAS,
help='Nominates a database to reset. Defaults to the "default" database.',
)
def handle_app_config(self, app_config, **options):
app_label = app_config.label
database = options['database']
interactive = options['interactive']
db_name = connections[database].settings_dict['NAME']
confirm = (ask_confirmation(app_label, db_name)
if interactive else 'yes')
if confirm == 'yes':
for model in app_config.get_models():
model.objects.using(database).all().delete()
self.stdout.write('Reset done.\n')
else:
self.stdout.write("Reset cancelled.\n")
def ask_confirmation(app_label, db_name):
return input("""You have requested a reset of the application {app_label}.
This will IRREVERSIBLY DESTROY all data related to the app currently in
the {db_name} database, and return each table to empty state.
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: """.format(**locals()))
Copy it to app/management/commands folder in any of your apps folders and run it with
./manage.py app_db_tables_reset any_installed_app_name
Ready-made package
The command is available in the django_commands package, you can install it with
pip install git+http://github.com/mrts/django-commands.git
and add it to INSTALLED_APPS to activate the command.
Tested with Django 1.9, it may or may not work with 1.8.
I am unable to use syncdb because my app uses some MySQL views. I have run manage.py sqlall <app>, but this does not output the SQL for django_content_type table or the auth_permission tables. I have also had a look into south and django evolution, but they both require syncdb, and I'm not sure they would help anyway.
I have manually added some models to the tables, but this is getting frustrating, and having installed the dbsettings app I am unsure of what I now need to enter.
Does anyone know of a way to get manage.py (or something else) to output the SQL for these tables and their contents?
Thanks.
Having done a bit more digging, I found these:
Fixing the auth_permission table after renaming a model in Django and manage.py sql command for django models - Django.
These output the tables, but not the data:
python manage.py sql auth
python manage.py sql admin
But this gets a lot closer. In the end I managed it with the following:
from django.contrib.auth.management import create_permissions
from django.db.models import get_apps
for app in get_apps():
create_permissions(app, None, 2)
from django.contrib.contenttypes.management import update_all_contenttypes
update_all_contenttypes(interactive=True)
This adds all the permissions and then all the content types which are needed. interactive=True means that it asks you if you want to remove stale content types.
#hajamie solution works for older supported version, taking a hint, below is what worked for me!
django = 1.9.7
from django.contrib.auth.management import create_permissions
from django.contrib.auth.models import Permission
from django.apps import apps
def fix_user_permission():
"""
run this method via shell whenever any amendments in any of the tables is made
"""
print "fixing user permissions"
# delete pre-existing user permission
Permission.objects.all().delete()
apps.models_module = True
create_permissions(apps, verbosity=0)
apps.models_module = None
print "process completed - fixed user permissions"
The easiest solution I found is to install Django Extensions, add it to settings.INSTALLED_APPS and run:
manage.py update_permissions
My project is in early development. I frequently delete the database and run manage.py syncdb to set up my app from scratch.
Unfortunately, this always pops up:
You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no):
Then you have supply a username, valid email adress and password. This is tedious. I'm getting tired of typing test\nx#x.com\ntest\ntest\n.
How can I automatically skip this step and create a user programatically when running manage.py syncdb ?
I know the question has been answered already but ...
A Much simpler approach is to dump the auth module data into a json file once the superuser has been created:
./manage.py dumpdata --indent=2 auth > initial_data.json
You can also dump the sessions data:
./manage.py dumpdata --indent=2 sessions
You can then append the session info to the auth module dump (and probably increase the expire_date so it does not expire... ever ;-).
From then, you can use
/manage.py syncdb --noinput
to load the superuser and his session when creating the db with no interactive prompt asking you about a superuser.
Instead of deleting your entire database, just delete the tables of your app before running the syncdb
This will accomplish it for you in a single line (per app):
python manage.py sqlclear appname | python manage.py dbshell
The first command will look at your app and generate the required SQL to drop the tables. This output is then piped to the dbshell to execute it.
After its done, run your syncdb to recreate the tables:
python manage.py syncdb
The key is to use --noinput at the time of syncdb & then use this one liner to create superuser
echo "from django.contrib.auth.models import User; User.objects.create_superuser('myadmin', 'myemail#example.com', 'hunter2')" | python manage.py shell
Credit : http://source.mihelac.org/2009/10/23/django-avoiding-typing-password-for-superuser/
If you want the ability — as I do — to really start with a fresh database without getting asked that superuser question, then you can just de-register the signal handler that asks that question. Check out the very bottom of the file:
django/contrib/auth/management/__init__.py
to see how the registration of the superuser function gets performed. I found that I could reverse this registration, and never get asked the question during "syncdb", if I placed this code in my "models.py":
from django.db.models import signals
from django.contrib.auth.management import create_superuser
from django.contrib.auth import models as auth_app
# Prevent interactive question about wanting a superuser created. (This
# code has to go in this otherwise empty "models" module so that it gets
# processed by the "syncdb" command during database creation.)
signals.post_syncdb.disconnect(
create_superuser,
sender=auth_app,
dispatch_uid = "django.contrib.auth.management.create_superuser")
I am not sure how to guarantee that this code gets run after the Django code that does the registration. I had thought that it would depend on whether your app or the django.contrib.auth app gets mentioned first in INSTALLED_APPS, but it seems to work for me regardless of the order I put them in. Maybe they are done alphabetically and I'm lucky that my app's name starts with a letter later than "d"? Or is Django just smart enough to do its own stuff first, then mine in case I want to muck with their settings? Let me know if you find out. :-)
I've overcome this feature using south
Its a must have for any django developer.
South is a tool designed to help migrate changes over to the live site without destroying information or database structure. The resulting changes can be tracked by south and using the generated python files - can perform the same actions on an alternative database.
During development, I use this tool to git track my database changes - and to make a change to the database without the need to destroy it first.
easy_install south
Add 'south' to your installed apps
Proposing first time run of south on an app.
$ python manage.py schemamigration appname --init
This will initiate schema detection on that app.
$ python manage.py migrate appname
This will apply the model changes
The database will have the new models.
Changing a model after the first run
$ python manage.py schemamigration appname --auto
$ python manage.py migrate appname
Models will have changed - data is not destroyed.
Plus south does much more...
Note: since version 1.7 syncdb command is deprecated. Use migrate instead.
Also Django 1.7 introduced AppConfig as means of customizing applications' initialization process.
Thus since Django 1.7 the simplest way to achieve what you want is to employ an AppConfig's subclass.
Let say, you happen to have your own example_app that is added to your INSTALLED_APPS and you want to create and admin user with admin password whenever you run ./manage.py migrate from scratch. I also assume that automatic admin user creation is required only in dev environment - not in production.
Add the following code to example_app/apps.py
# example_app/apps.py
from django.apps import AppConfig
from django.conf import settings
from django.db.models.signals import post_migrate
from django.contrib.auth.apps import AuthConfig
USERNAME = "admin"
PASSWORD = "admin"
def create_test_user(sender, **kwargs):
if not settings.DEBUG:
return
if not isinstance(sender, AuthConfig):
return
from django.contrib.auth.models import User
manager = User.objects
try:
manager.get(username=USERNAME)
except User.DoesNotExist:
manager.create_superuser(USERNAME, 'x#x.com', PASSWORD)
class ExampleAppConfig(AppConfig):
name = __package__
def ready(self):
post_migrate.connect(create_test_user)
Also add the following reference to the app configuration inside apps example_app/__init__.py:
# example_app/__init__.py
default_app_config = 'example_app.apps.ExampleAppConfig'
Where the default_app_config is a string Python path to the AppConfig subclass as mentioned here.
The manage.py reset command will reset your database without destroying your created super user. Data does however need to be re-imported.
You can use django-finalware to do this for you. Just add finalware to your INSTALLED_APPS and include the following in your settings.py:
SITE_SUPERUSER_USERNAME = 'myadmin'
SITE_SUPERUSER_EMAIL = 'myadmin#example.com'
SITE_SUPERUSER_PASSWORD = 'mypass' # this can be set from a secret file.
# optional object id. Ensures that the superuser id is not set to `1`.
# you can use this as a simple security feature
SITE_SUPERUSER_ID = '343'
Then just run ./manage.py syncdb (Django <1.7) or ./manage.py migrate (Django >= 1.7), and it will automatically create a superuser or update the existing one for you.
You are never prompted to created a superuser anymore.
Since Django 1.7 the suggested way of populating the database is through data migrations. To create a data migration for creating the admin you should first create an empty migration:
./manage.py makemigrations --empty myapp --name create-superuser
This will create an empty migration in myapp/migrations/000x__create-superuser.py. Edit the file to make it look like this:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import migrations, models
from django.contrib.auth.models import User
def create_superuser(apps, schema_editor):
User.objects.create_superuser(username='myadmin', password='mypassword', email='myemail#gmail.com')
class Migration(migrations.Migration):
dependencies = [('myapp', '000y_my-previous-migration-file'),]
operations = [migrations.RunPython(create_superuser)]
I have resolved creating a python script like this one to reset all my stuff [updated version][1.8 too]:
import os
import sys
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "main.settings.dev")
from django.conf import settings
from django.core import management
from django import get_version
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
if PROJECT_ROOT not in sys.path:
sys.path.append(PROJECT_ROOT)
yn = raw_input('Are you sure you want to reset everything? (y/n) ')
if yn == 'y':
# Drops the db / creates the db
if settings.DATABASES['default']['ENGINE'].find('mysql') != -1:
os.system('mysqladmin -uroot -pIronlord0 -f drop db')
os.system('mysqladmin -uroot -pIronlord0 -f create db')
elif settings.DATABASES['default']['ENGINE'].find('psycopg2') != -1:
os.system('psql -U postgres -c "DROP DATABASE db"')
os.system('psql -U postgres -c "CREATE DATABASE db WITH OWNER = admin"')
elif settings.DATABASES['default']['ENGINE'].find('sqlite3') != -1:
try:
os.remove(os.path.join(PROJECT_ROOT, 'data.db'))
except:
pass
# Getting application handle here otherwise db gets allocated and it can not be destroyed.
if get_version() > '1.6.10':
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
management.call_command('syncdb', interactive=False)
# Creates admin/password
from django.contrib.auth.management.commands import changepassword
management.call_command('createsuperuser', interactive=False, username="admin", email="xxx#example.com")
command = changepassword.Command()
command._get_pass = lambda *args: 'password'
if get_version() >= '1.8':
command.execute(username="admin")
else:
command.execute("admin")
# Creates the default site entry
from django.contrib.sites.models import Site
site = Site.objects.get_current()
site.domain = 'www.example.com'
site.name = ' xxx '
site.save()
it works like a charm!
P.S.: Be sure to stop your (testing) server where above db is in charge before running this script!
Take a look at the dumpdata management command. For instance:
python manage.py dumpdata > initial_data.json
If this file, called a fixture, is named initial_data (.xml or .json), then the syncdb command will pick it up and populate your tables accordingly. It will still ask you if you want to create a user, but I believe you may safely answer "no", after which point it will populate the database based on your fixture.
More info on this can be found in the docs.
Developing with sqlite.
Clear database by deleting file.
Load admin from fixtures.
change manage.py (django 1.4):
# hack to prevent admin promt
if len(sys.argv) == 2 and sys.argv[1] == 'syncdb':
sys.argv.append('--noinput')
My solution to this was to just not delete that auth tables when wiping out my database.
If you prefer to type initializing code direct into python source file, this code modified manage.py might help (and thanks for Cjkjvfnby's little code!):
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
# set your django setting module here
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings")
from django.core.management import execute_from_command_line
# hack to prevent admin prompt
if len(sys.argv) == 2 and sys.argv[1] == 'syncdb':
sys.argv.append('--noinput')
execute_from_command_line(sys.argv)
# additional process for creation additional user, misc data, and anything
for arg in sys.argv:
# if syncdb occurs and users don't exist, create them
if arg.lower() == 'syncdb':
print 'syncdb post process...'
from django.contrib.auth.models import User
admin_id = 'admin'
admin_email = 'superuser#mail.com'
admin_password = 'superuser_password'
additional_users = [
['tempuser', 'user_email#mail.com', 'tempuser_password']
]
# admin exists?
user_list = User.objects.filter(username=admin_id)
if len(user_list) == 0:
print 'create superuser: ' + admin_id
new_admin = User.objects.create_superuser(admin_id, admin_email, admin_password)
# additional user exists?
for additional_user in additional_users:
user_list = User.objects.filter(username=additional_user[0])
if len(user_list) == 0:
print 'create additional user: ' + additional_user[0]
new_admin = User.objects.create_user(additional_user[0], additional_user[1], additional_user[2])
# any other data
I'm just showing the user creation code here, but you can enhance this code more as you want.
I'm using sqlite as a dev database. After changing model classes, just drop the corresponding tables with sqlite manager (a firefox plugin, open to inspect the data anyways) and run manage.py syncdb to recreate what's missing.