Why does running sqlclear and syncdb not drop sqlite3 tables - django

I want to drop some tables I created for the polls app from the Django tutorials. I am using Django 1.6.5.
I start by turning off the server and then running
manage.py sqlclear
This prints out
BEGIN;
DROP TABLE "polls_choice";
DROP TABLE "polls_poll";
COMMIT;
I then run
manage.py syncdb
When I start the server, go to the admin panel, and click on the models, the values are all still there. Why is this happening? It seems like if I'm dropping those tables, at the very least any existing values should be gone.

According to the documentation, the manage.py sqlclear command only prints queries to execute to clear the database. You need to execute them by yourself.
You can also use manage.py flush to get the empty database (read carefully the documentation, it is actually executing some handlers and adding initial fixtures if you have any)
django-admin.py sqlclear
Prints the DROP TABLE SQL statements for the given app name(s).
The --database option can be used to specify the database for which to print the SQL.

you need to enter here:
python manage.py dbshell
and paste what sqlclear printed out

Related

Django Programming error column does not exist even after running migrations

I run python manage.py makemigrations and I get:
No changes detected
Then, python manage.py migrate and I get:
No migrations to apply.
Then, I try to push the changes to production:
git push heroku master
Everything up-to-date
Then, in production, I repeat the command:
heroku run python manage.py migrate
No migrations to apply.
Just in case, I run makemigrations in production:
heroku run python manage.py makemigrations
No changes detected
WHY then I get a
ProgrammingError at ....
column .... does not exist
"No changes detected" means the database is coherent with the code.
How can I debug this?ยก?
I got the same problem (column not exist) but when I try to run migrate not with makemigrations (it is the same issue I believe)
Cause: I removed the migration files and replaced them with single pretending intial migration file 0001 before running the migration for the last change
Solution:
Drop tables involved in that migration of that app (consider a backup workaround if any)
Delete the rows responsible of the migration of that app from the table django_migrations in which migrations are recorded, This is how Django knows which migrations have been applied and which still need to be applied.
And here is how solve this problem:
log in as postgres user (my user is called posgres):
sudo -i -u postgres
Open an sql terminal and connect to your database:
psql -d database_name
List your table and spot the tables related to that app:
\dt
Drop them (consider drop order with relations):
DROP TABLE tablename ;
List migration record, you will see migrations applied classified like so:
id | app | name | applied
--+------+--------+---------+
SELECT * FROM django_migrations;
Delete rows of migrations of that app (you can delete by id or by app, with app don't forget 'quotes'):
DELETE FROM django_migrations WHERE app='yourapp';
log out and run your migrations merely (maybe run makemigrations in your case):
python manage.py migrate --settings=your.settings.module_if_any
Note: it is possible that in your case will not have to drop all the tables of that app and not all the migrations, just the ones of the models causing the problem.
I wish this can help.
Django migrations are recorded in your database under the 'django_migrations' table. This is how Django knows which migrations have been applied and which still need to be applied.
Have a look at django_migrations table in your DB. It may be that something went wrong when your migration was applied. So, delete the row in the table which has the migration file name that is related to that column that 'does not exist'. Then, try to re-run a migration.
Here's what i tried and it worked:
Go and add manually the column to your table
run python manage.py makemigrations
go back drop that column you added
run python manage.py migrate
I had a similar issue - the error message appeared when I clicked on the model on the django-admin site. I solved it by commenting out the field in models.py, then running migrations. Following this I uncommented the field and re ran the migrations. After that the error message disappeared.
My case might be a bit obscure, but if it helps someone, it is worth documenting here.
I was calling a function in one of my migrations, which imported a Model of said migration regularly, i.e.
from myApp.models import ModelX
The only way models should be imported in migrations would be using e.g. RunPython:
def myFunc(apps, schema_editor):
MyModel = apps.get_model('myApp 'MyModel')
and then calling that function like so:
class Migration(migrations.Migration):
operations = [
migrations.RunPython(initialize_mhs, reverse_code=migrations.RunPython.noop),
]
Additionally the original import worked until I modified the model in a later migration, making this error harder to locate.
So, I always run into this sort of problem, so today I decided to try and work it out at the database level. Thing is, I changed a model field name which Django didn't bother reflecting in a migration file. I only found out later when I ran into problems. I later looked at the migration files and discovered there was no migration for that change. But I didn't notice because I made other changes as well, so once I saw a migration file I was happy.
My advice. Create migration for each change one at a time. That way you get to see if it happened or not.
So here's my working through it in MySQL.
open mysql console.
show databases; # see all my dbs. I deleted a few
drop database <db-name>; # if needed
use <db-name>; # the database name for your django project
show tables; # see all tables in the database
DESCRIBE <table-name>; # shows columns in the database
SHOW COLUMNS FROM <db-name>; # same thing as above
ALTER TABLE <table-name> CHANGE <old-column-name> <new-column-name> <col-type>; # now I manually updated my column name
If you're using postgresql, just google the corresponding commands.
The issue was in the Models for me, for some reason Django was adding '_id' to the end of my Foreign Key column. I had to explicitly set the related named to the Foreign Key. Here 'Cards' is the parent table and 'Prices' is the child table.
class Cards(models.Model):
unique_id = models.CharField(primary_key=True, max_length=45)
name = models.CharField(max_length=225)
class Prices(models.Model):
unique_id = models.ForeignKey(Cards, models.DO_NOTHING)
Works after changing to:
class Cards(models.Model):
unique_id = models.CharField(primary_key=True, max_length=45)
name = models.CharField(max_length=225)
class Prices(models.Model):
unique_id = models.ForeignKey(Cards, models.DO_NOTHING, db_column='unique_id')
When I get this error, my extreme way to solve it is to reset my database:
Reset your database
For Postgresql on Heroku:
Heroku > your_app > Resources > database > add-ons > click on your database and open it
For postgresql
settings > Reset database
Delete all files in your_app > migrations > __pycache__ except __init.py__
Delete all files in your_app > migrations except __pycache__ folder and __init.py__
Then run:
python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser
type in to create your superuser, then run:
python manage.py makemigrations
python manage.py migrate
python manage.py
If you are able to inspect your models from your admin section, then it should be all okay now.
Just remove corresponding row migrations for that model in 'django_migrations' model in database.
And re run python manage.py migrate app_name
I tried all these answers with not much luck! What I did to have this problem solved with no harm was to go back through the migration files and find where the actual model was being created for the first time then manually add the field (in the column not being existed error message). Until if you run makemigrations --dry-run you get/see "No changes detected" and that worked. Basically, in my case, I had to carefully take my desired db changes back in time in proper migration file, rather creating a new migration now at the end of migration dependency chain.
Open the latest py file created after running the makemigrations command inside migrations folder of that particular app.
Inside class Migration there is a list attribute called 'operations'.
Remove the particular elements migrations.RemoveField(...).
Save and run python manage.py migrate.
A easier solution to the problem is to make your models exactly like it is in the migration first. and run python manage.py migrate.
Then revert those changes
Run
python manage.py makemigrations
python manage.py migrate
To check for which migrations are applied and which are not, use -:
python manage.py showmigrations
I solved a similar problem by deleting all migrations files (Don't forget to make a backup) and python manage.py makemigrations all of them into one clean file in development and pulling new files on the server. Before then I had dropped existing tables on the PostgreSQL.
I was getting this error for some reason when Django was looking for a column of type ForeignKey named category_id when the actual name in the database was category. I tried every Django solution I could imagine (renaming field, explicitly setting column name, etc.). I didn't want to drop tables or rows as this was a production database. The solution was simply to rename the column manually using SQL. In my case:
ALTER TABLE table_name
RENAME COLUMN category TO category_id;
Make sure you backup your database, ensure this won't break any other applications consuming that particular table, and consider having a fallback column if necessary.
What helped me in the end was simply dropping the database and creating it again as well as deleting all migrations files (including cache). (only removing migrations file didn't work for me at all)
sudo su - postgres
psql
DROP DATABASE 'yourdatabase';
CREATE DATABASE 'yourdatabase';
GRANT ALL PRIVILEGES ON DATABASE 'yourdatabase' to 'yourdjangouser';
then just
python manage.py makemigrations
python manage.py migrate
python manage.py runserver
If you're in development and you make some examples of data that's not important, this step is beneficial for me: just flush your data, make migrations, and migrate:
python manage.py flush
python manage.py makemigrations
python manage.py migrate
After that, you may create a new database from scratch, I hope this information was helpful.
Solved this issue by running
python manage.py migrate
in Heroku Bash shell

Django - view sql query without publishing migrations

When I make changes to some models, I want to view the SQL the django would be running to implement those changes on the DB.
The normal way of doing this would be to do 'makemigrations appname'. This will generate a migration, let's say, - '0001_someName.py'. Then one can do 'sqlmigrate 0001_someName.py'
But I want to view the sql directly, without having to create that intermediate migration. Can this be done?
Use the sqlmigrate command from manage.py.
python manage.py sqlmigrate <appname> <migration number eg. 0001 or 0004>
will display the SQL statements for a specific migration of the app.
Django does not provide that option. You can always create the migration, run sqlmigrate, and delete the migration file. As long as it isn't applied with migrate, nothing will happen.
These existing answers are not enough, as I found out trying to follow them. First detect and make the migration script for your app:
manage.py makemigrations app
Make note of the four-digit migration number which starts the filename. Then print the SQL with:
manage.py sqlmigrate app 0002 # <-- number here
When finished, remove the file before it gets run or committed:
rm api/migrations/0002_auto_8675309.py
Run
python manage.py sql <appname>
-- Prints the CREATE TABLE SQL statements for the given app name(s).
python manage.py sqlall <appname>
-- Prints the CREATE TABLE and initial-data SQL statements for the given app name(s).
You'll find detail documentation here.
https://docs.djangoproject.com/en/1.8/ref/django-admin/

How to "reset" postgresql database?

I am trying to move my django project into a production environment and in doing so I switched from using sqlite to postgres. In my development environment, whenever I made changes to my models or anything that would significantly change how the database was setup, I would literally just drag my sqlite file to the trash and just run syncdb to create a new empty one (probably bad practice). Now that I am using postgres, I am wanting to do the same thing without actually deleting the database. Basically I was wondering if there was a way to completely empty it or clear it out and then just run syncdb and start over?
I also welcome any alternative suggestions that might lead me down the right path, I'm very new to this.
You can use flush. Just run this command:
python manage.py flush
First if you have initial data in your database you can use dumbpdata command:
python manage.py dumpdata > initial_data.json
For specific app run:
python manage.py dumpdata <app_name> > initial_data.json
Second run the flush command to clean your database:
python manage.py flush
Third and last, run loaddata command to load the initial data into your database and create superuser by running createsuperuser command
python manage.py loaddata initial_data.json
python manage.py createsuperuser
In case flush does not work, you can drop the whole database.
Go to windows command line.
If server is 'postgres' and db name is 'mydb', run:
C:\> psql -U postgres
You will see a postgres-# prompt. Next is to close connections running the following:
SELECT * FROM pg_stat_activity WHERE pg_stat_activity.datname='mydb';
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'mydb';
Drop database once for all: DROP DATABASE mydb;

What is the django command to delete all tables?

Are there django commands that
A. Delete all tables
B. delete all data in all tables
C. Create all tables as defined in the model?
I cannot find these right now!
And by commands i mean those little things that are like
runserver
etc
A. Delete all tables
manage.py sqlclear will print the sql statement to drop all tables
B. delete all data in all tables
manage.py flush returns the database to the state it was in immediately after syncdb was executed
C. Create all tables as defined in the model?
manage.py syncdb Creates the database tables for all apps in INSTALLED_APPS whose tables have not already been created.
See this page for a reference of all commands: https://docs.djangoproject.com/en/dev/ref/django-admin/
But you should definitely look into using south as someone already mentioned. It's the best way to manage your database.
N.B: syncdb is deprecated in favour of migrate, since Django 1.7.
If you have the client libraries installed for your database, you can run this:
python manage.py sqlflush | python manage.py dbshell
This doesn't drop the tables, but truncates them.
There isn't a command that does the it all in one go, but this "one liner" will drop all the tables and then re-create them. It would only work if you were running on a system that provides these utilities at the shell.
echo 'from django.conf import settings; print settings.INSTALLED_APPS; quit();' | python manage.py shell --plain 2>&1 | tail -n1 | sed -r "s|^.*\((.*)\).*$|\1|; s|[',]| |g; s|django\.contrib\.||g" | xargs python manage.py sqlclear | python manage.py dbshell && python manage.py syncdb
Neither manage.py sqlclear nor manage.py reset is capable of dropping all tables at once, both require an appname parameter.
You should take a look at Django Extensions, it gives you access to manage.py reset_db as well as many other useful management commands.
I recommend using django-south. It allows you to sync your models to your database not just when you add a field, but also when you delete a field/model. I really find it to be an essential component of building a Django site. Once installed, you could run a command like:
./manage.py migrate app zero
You can learn more here: http://south.aeracode.org/docs/about.html
And a simpler oneliner to drop all the tables for django 1.5+:
python2 manage.py sqlflush | sed 's/TRUNCATE/DROP TABLE/g'| python2 manage.py dbshell
I was able todrop my tables. Run
python manage.py sqlclear appname
Take note of the commands given. Then run
python manage.py dbshell
Add the commands given in the first step. line by line. When you are done, type '.exit' (for SQlite3). Resync your DB and you should be good to go. To be sure, check the tables with:
python manage.py shell
>>> from yourapp import yourclasses
>>> yourviews.objects.all()
it should return a []. I hope this helps.
A implies B, right?
For A, see How to drop all tables from the database with manage.py CLI in Django?
For C,
python manage.py syncdb
Ofcourse for smart data migration I go with what #bento mentioned: django-south

Django flush vs sqlclear & syncdb

Can anyone tell if there is a difference between
>manage.py flush # or reset
and
>manage.py sqlclear appname | python manage.py dbshell
>manage.py syncdb
flush will truncate (delete data)
sqlclear will drop (delete table, thus data too)
=> if you have structural modifications in your db, you have to do sqlclear (but better use south)
Update:
South has been deprecated.
From Django 1.7 upwards, migrations are built into the core of Django. If you are running a previous version, you can use South.
Official docs for
flush and
sqlclear
Flush carries out the SQL Drops on the entire db, sqlflush only prints out the SQL that flush would actual run (again on the entire db). sqlclear prints out SQL Drops for a particular app or apps. Both flush and sqlflush/dbshell/syncdb will install fixtures.