Context
Some steps of my Continuos Integration procedure are:
start Postgres docker container
run Django tests
When manage.py test --noinput command is runned it:
creates a new test_xxx database (drop if exists)
runs the founded migrations against it
runs the founded set of tests
Into the tests that need to fetch data from the database are configured a set of fixtures that will be loaded automatically in the test_xxx db.
Problem
Some migrations need the Postgres hstore extension, in fact i'm getting this error:
django.db.utils.ProgrammingError: type "hstore" does not exist
Question
How can i enable the hstore extension?
In development and other envs it was set up with CREATE EXTENSION IF NOT EXISTS hstore; but here can't be manually set.
Is possible to define a "migration zero" with the hstore creation? Anyway i don't like this approach.
I've found that it should be theorically possible to listen to the pre_migrate signal, and it will be the sweet spot, but before make things more complex i'd like to search for an easier, more direct solution.
EDIT: in this particular case i must use Django 1.8, since 1.11 is possible to define a template from which create the test db, so is possible to define a template with the hstore and problem solved.
The final workaround until the Django update is:
when start the Postgre container add an init.sql file with the test_xxx database creation and hstore extension
when launching the django tests, instead of using the --noinput flag, use the --keepdb one so the db isn't recreated
Related
This is a Heroku-specific issue with a Django project 1.11.24 running Python 3.6.5 and a Heroku postgres database.
During testing of two different branches during development, different conflicting migration files were deployed at different times to the Heroku server. We recognized this, and have now merged the migrations, but the order the Heroku psql db schema was migrated is out of order with the current migration files.
As a result, specific tables already exist, so on deploy applying the updated merged migration files errs with:
psycopg2.errors.DuplicateTable: relation "table_foo" already exists
In heroku run python manage.py showmigrations -a appname all of the migrations are shown as having run.
We've followed Heroku's docs and done the following:
Rolled back the app itself to before when the conflicting migrations took place and were run (https://blog.heroku.com/releases-and-rollbacks)
Rolled back the postgres db itself to a datetime before when the conflicting migrations took place and were run (https://devcenter.heroku.com/articles/heroku-postgres-rollback)
However, despite both app and db rollbacks, when we check the db tables themselves in the rollback in pql shell with \dt, the table causing the DuplicateTable err still exists, so the db rollback doesn't actually seem to effect the django_migrations table.
It's Heroku, so we can't fake the migrations.
We could attempt to drop the specific db tables that already exist (or drop the entire db, since it's a test server), but that seems like bad practice. Is there any other way to address this in Heroku? thanks
I eventually fixed this by manually modifying migration files to align with the schema dependency order that was established. Very unsatisfying fix, wish Heroku offered a better solution for this (or a longer postgres database rollback window)
I have a flask setup with development & staging environments.
Now I want to add a production environment wit a production database.
I have troubles integrating the new database into flask-migrations.
I did these steps:
created fresh postgres DB
ran db.create_all() from the flask app
(resulting in a DB reflecting the latest version of the data model)
now all flask-migrate commands have errors
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "testfield" of relation "customer_feedback" already exists,
because flask migrate
seems to think it needs to apply all migrations that have been
created until today. But they are not necessary because DB is already fully reflecting models.py.
How can I convince flask-migrate to accept the current state as fully migrated?
Or whats the standard workflow for this?
In other words:
I am coming from Django, where the migrate command creates and updates the model if necessary when adding a blank DB. How should it be done with flask?
You need to tell flask migrate that db has already been created and all requirements are already fulfilled. Try following command -
flask db stamp head
This will tell flask migrate not to attempt to add anything.
I need to know if getting an error while running python manage.py migrate means my database will remain in the same state it was before running the migrate command.
I'm trying to implement migrations as part of a CI system and it would be good to know if I need to do some kind of rollback if the migrations fails.
As the documentation explains, it depends on the database.
PostgreSQL can use transactions for schema alteration operations, so Django does so, and rolls back in case of failure. But MySQL does not support this.
I have a app that is deployed to Heroku, and I'd like to be able to run the test suite post-deployment on the target environment. I am using the Heroku Postgres add-on, which means that I have access to a single database only. I have no rights to create new databases, which in turn means that the standard Django test command fails, as it can't create the test_* database.
$ heroku run python manage.py test
Running `python manage.py test` attached to terminal... up, run.9362
Creating test database for alias 'default'...
Got an error creating the test database: permission denied to create database
Is there any way around this?
Turns out I was in the wrong. I was not testing what I thought was being tested... Since Heroku's Routing Mesh was sending requests to different servers, the LiveServerTestCase was starting a web server on one machine and Selenium was connecting to other machines altogether.
By updating the Heroku Procfile to:
web: python src/manage.py test --liveserver=0.0.0.0:$PORT
overriding the DATABASES setting to point to the test database, and customizing the test suite runner linked to below (the same idea still holds: override setup_databases so that it only drops/re-creates tables, not the entire database), I was able to run remote tests. But this is even more hacky/painful/inelegant. Still looking for something better! Sorry about the confusion.
(updated answer below)
Here are the steps that worked for me:
Create an additional, free Postgres database using the Heroku toolbelt
heroku addons:add heroku-postgresql:dev
Use the HerokuTestSuiteRunner class which you'll find here.
This custom test runner requires that you define a TEST_DATABASES setting which follows the typical DATABASES format. For instance:
TEST_DATABASES = {
'default': dj_database_url.config(env='TEST_DATABASE_URL')
}
Then, have the TEST_RUNNER setting be a Python path to wherever HerokuTestSuiteRunner can be found.
You should now be able to run Django tests on Heroku using the given database. This is very much a quick hack... Let me know how it could be improved / made less hackish. Enjoy!
(original answer below)
A few relevant solutions have been discussed here. As you can read in the Django docs, "[w]hen using the SQLite database engine, the tests will by default use an in-memory database".
Although this doesn't thoroughly test the database engine you're using on Heroku (I'm still on the lookout for a solution that does that), setting the database engine to SQLite will at least allow you to run your tests.
See the above-linked StackOverflow question for some pointers. There are at least two ways out: testing if 'test' in sys.argv before forcing SQLite as the database engine, or having a dedicated settings file used in testing, which you can then pass to django manage.py test using the --settings option.
Starting with version 1.8, Django now has an option called keepdb, which allows for the same database to be reused during tests.
The --keepdb option can be used to preserve the test database between test runs.
This has the advantage of skipping both the create and destroy actions which can greatly decrease the time to run tests, especially those in a large test suite.
If the test database does not exist, it will be created on the first run and then preserved for each subsequent run.
Any unapplied migrations will also be applied to the test database before running the test suite.
Since it also allows for the test database to exist prior to running tests, you can simply add a new Postgres Heroku instance to your dyno and configure the tests to use that particular database.
Bonus : you can also use the failfast option, which exits as soon as your first test crashes, so that you don't have to wait for all tests to complete.
However, if you are deploying things to Heroku and you are using Heroku Pipelines, an even better option is available : Heroku CI.
I have some existing SQL statements that I'd like to use as a custom hook after the CREATE TABLE command, in [myapp]/sql/[model-name].sql.
My app is in INSTALLED_APPS. I see listed if I run ./manage.py sql.
My custom hook is found; I see the SQL statements output if I run any of the following:
./manage.py sqlall <myapp>
./manage.py sqlcustom <myapp>
./manage.py sql <myapp>
I'm using postgres 9.x on my mac.
If I psql to that same database (with no user) and copy them from the .sql file and paste them into the psql command input, they all work... so I believe they're valid SQL understood by postgres. These are all pretty simple INSERT statements (fixtures addressed below).
However, if I run ./manage.py syncdb those statements are either not run, or they are ignored or silent errors happen; all I know is that the new rows do not appear in the database. I am tailing the postgres log file and nothing is logged when I run syncdb, so I don't know if it's not finding my .sql file, or parsing it and finding some error before it gets to the database.
I have created a .json file, for fixtures, with the equivalent of those statements, and ./manage.py loaddata <path-to-json-file> works correctly: my site now shows those values in the database. This makes me believe that my settings file is correct and the database I'm writing to inside postgres is set correctly, and I have write permissions when I run ./manage.py.
I saw in some other post that the django documentation is wrong and I should put the custom hook in the 'models' directory, but I don't know if that's right; if sqlall and sqlcustom find my hook, shouldn't syncdb find it? Also I don't (yet) have a models directory and may not need it.
For various reasons I'd rather not use JSON format, but if I have to I will... however I've invested so much time in the .sql format I really want to know what's going on (and I've seen enough existing related questions that this might help others).
I believe I found it, although it's based on behaviors not any real research. I simply changed 'tile' to 'tilexx' everywhere and it worked. This django-project post indicates that if there is some sort of python class name conflict the custom SQL won't be executed... and 'tile' is a pretty common thing.
So the answer is to change the name of my class to something a bit more unique.
I've been searching for an answer to a similar problem, trying to initialize an sqlite database with data I dumped from a Flask application I'm porting to Django. Like the OP, all of the following did the right thing:
./manage.py sqlall <myapp> ./manage.py sqlcustom <myapp> ./manage.py sql <myapp>
However, the insert statements in myapp/sql/myapp.sql were not being run. A careful reading of the documentation revealed this clue:
Django provides a hook for passing the database arbitrary SQL that’s executed just after the CREATE TABLE statements when you run syncdb.
(emphasis added)
The issue is that I had already registered my models and run syncdb, so the table in question already existed in the database, although it held no data yet. I deduce that because of this, the CREATE TABLE statement was not being run on subsequent executions of syncdb, and therefore, my custom sql could not be run after that statement. The solution was to DROP table table_name and then run syncdb again, at which point my custom sql was run.