I got errors running pytest on Travis CI but have no idea how to fix it. When I run the command docker-compose run --rm api sh -c "pytest && flake8" on Docker in my local, all the tests pass. Could anyone give me a hint? I have some fixtures on conftest.py as well.
A part of the error information
api/tests/test_order_items.py EEEEE [ 45%]
api/tests/test_orders.py EEEEEE [100%]
==================================== ERRORS ====================================
__________ ERROR at setup of TestOrderItemModel.test_list_order_items __________
self = <django.db.backends.utils.CursorWrapper object at 0x7efe0e94d3d0>
sql = 'SELECT "orders_orderitem"."id", "orders_orderitem"."order_id", "orders_orderitem"."pizza_type", "orders_orderitem"."pizza_size", "orders_orderitem"."quantity" FROM "orders_orderitem" ORDER BY "orders_orderitem"."id" ASC'
params = ()
ignored_wrapper_args = (False, {'connection': <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7efe0fbb4fd0>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x7efe0e94d3d0>})
def _execute(self, sql, params, *ignored_wrapper_args):
self.db.validate_no_broken_transaction()
with self.db.wrap_database_errors:
if params is None:
return self.cursor.execute(sql)
else:
> return self.cursor.execute(sql, params)
E psycopg2.errors.UndefinedColumn: column orders_orderitem.order_id does not exist
E LINE 1: ...SOR WITH HOLD FOR SELECT "orders_orderitem"."id", "orders_or...
.travis.yml
language: python
python:
- "3.7"
services:
- docker
before_script: pip install docker-compose
script:
- docker-compose run --rm api sh -c "pytest && flake8"
Pipfile
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
flake8 = "==3.7.9"
autopep8 = "==1.4.4"
pytest = "==5.2.1"
pytest-django = "==3.6.0"
[packages]
django = "==2.2.7"
djangorestframework = "==3.10.3"
psycopg2 = "==2.8.4"
[requires]
python_version = "3.7"
Related
Fairly new to Docker, I am trying to add the execution of a custom sql script (triggers and functions) to Django's migration process and I am starting to feel a bit lost. Overall, What I am trying to achieve follows this pretty clear tutorial. In this tutorial, migrations are achieved by the execution of an entry point script. In the Dockerfile:
# run entrypoint.sh
ENTRYPOINT ["/usr/src/my_app/entrypoint.sh"]
Here is the entrypoint.sh:
#!/bin/sh
if [ "$DATABASE" = "postgres" ]
then
echo "Waiting for postgres..."
while ! nc -z $SQL_HOST $SQL_PORT; do
sleep 0.1
done
echo "PostgreSQL started"
fi
# tried several with and without combinations
python manage.py flush --no-input
python manage.py makemigrations my_app
python manage.py migrate
exec "$#"
So far so good. Turning to the question of integrating the execution of custom sql scripts in the migration process, most articles I read (this one for instance) recommend to create an empty migration to add the execution of sql statements. Here is what I have in
my_app/migrations/0001_initial_data.py
import os
from django.db import migrations, connection
def load_data_from_sql(filename):
file_path = os.path.join(os.path.dirname(__file__), '../sql/', filename)
sql_statement = open(file_path).read()
with connection.cursor() as cursor:
cursor.execute(sql_statement)
class Migration(migrations.Migration):
dependencies = [
('my_app', '0001_initial'),
]
operations = [
migrations.RunPython(load_data_from_sql('my_app_base.sql'))
]
As stated by dependencies, this step depends on the initial one (0001_initial.py):
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
initial = True
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name='Unpayed',
fields=[
etc etc
[The Issue] However, even when I try to manually migrate (docker-compose exec web python manage.py makemigrations my_app), I get the following error because the db in the postgresql container is empty:
File "/usr/src/my_app/my_app/migrations/0001_initial_data.py", line 21, in Migration
migrations.RunPython(load_data_from_sql('my_app_base.sql'))
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 82, in _execute
....
return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "auth_user" does not exist
[What I do not understand] However, when I log in the container, remove 0001_initial_data.py and run ./entrypoint.sh, everything works like a charm and tables are created. I can add 0001_initial_data.py manually later on, run entrypoint.sh angain and have my functions. Same when I remove this file before running docker-compose up -d --build: tables are created.
I feel like I am missing some obvious way and easier around trying integrate sql script migrations in this canonical way. All I need is this script to be run after 0001_initial migration is over. How would you do it?
[edit] docker-compose.yml:
version: '3.7'
services:
web:
build:
context: ./my_app
dockerfile: Dockerfile
command: python /usr/src/my_app/manage.py runserver 0.0.0.0:8000
volumes:
- ./my_app/:/usr/src/my_app/
ports:
- 8000:8000
environment:
- SECRET_KEY='o##xO=jrd=p0^17svmYpw!22-bnm3zz*%y(7=j+p*t%ei-4pi!'
- SQL_ENGINE=django.db.backends.postgresql
- SQL_DATABASE=postgres
- SQL_USER=postgres
- SQL_PASSWORD=N0tTh3D3favlTpAssw0rd
- SQL_HOST=db
- SQL_PORT=5432
depends_on:
- db
db:
image: postgres:10.5-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
volumes:
postgres_data:
django:2.2
python:3.7
I believe the issue has to do with you naming the migration file and manually making your dependencies with the same prefix "0001" The reason I say this is because when you do reverse migrations, you simply can just reference the prefix. IE if you wanted to go from your 7th migration to your 6th migration. The command looks like this python manage.py migrate my_app 0006 Either way, I would try deleting and creating a new migration file via python manage.py makemigrations my_app --empty and moving your code into that file. This should also write the dependencies for you.
The error message alongside the test you ran with adding he migration file after is indicative of the issue though. Some how initial migrations aren't running before the other one. I would also try dropping your DB as it may have persisted some bad state ./manage.py sqlflush
[The easiest way I could find] I simply disentangled django migrations from the creation of custom functions in the DB. Migration are run first so that the tables exists when creating the functions. Here is the entrypoint.sh
#!/bin/sh
if [ "$DATABASE" = "postgres" ]
then
echo "Waiting for postgres..."
while ! nc -z $SQL_HOST $SQL_PORT; do
sleep 0.1
done
echo "PostgreSQL started"
fi
python manage.py flush --no-input
python manage.py migrate
# add custom sql functions to db
cat my_app/sql/my_app_base.sql | python manage.py dbshell
python manage.py collectstatic --no-input
exec "$#"
Keep in mind that manage.py dbshell requires a postgresql-client to run. I just needed to add it in the Dockerfile:
# pull official base image
FROM python:3.7-alpine
...........
# install psycopg2
RUN apk update \
&& apk add --virtual build-deps gcc python3-dev musl-dev \
&& apk add postgresql-dev postgresql-client\
&& pip install psycopg2 \
&& apk del build-deps
There are a lot of django-related posts with "no such table" error but none are coming from django-user-accounts module.
I am getting this error
sqlite3.OperationalError: no such table: account_passwordhistory
, and here are the steps:
Installed django-user-accounts:
$ pip install django-user-accounts
Collecting django-user-accounts
Downloading https://files.pythonhosted.org/packages/0c/4f/40f76395324d98b7f8717aad7e08ad6f86ba2714eb956be6070e0402988c/django_user_accounts-2.0.3-py2.py3-none-any.whl (106kB)
100% |████████████████████████████████| 112kB 2.8MB/s
.
.
.
Installing collected packages: django-appconf, django-user-accounts
Successfully installed django-appconf-1.0.2 django-user-accounts-2.0.3
in settings.py added
INSTALLED_APPS = [ . . .
'django.contrib.sites',
'account' ]
SITE_ID = 1
MIDDLEWARE = [
.
.
.
'account.middleware.ExpiredPasswordMiddleware',
]
ACCOUNT_PASSWORD_USE_HISTORY = True
ACCOUNT_PASSWORD_EXPIRY = 60 # number of seconds
restarted server and ran : $ python manage.py user_password_history
I got:
Traceback (most recent call last):
File "/Users/ipozdnya/miniconda3/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute return self.cursor.execute(sql, params)
File "/Users/someuser/miniconda3/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: account_passwordhistory
I realize that account_passwordhistory did not get created at the time of install or some other step. $ python manage.py makemigrations states that No changes detected. Nothing in this doc tells me what to do about it: http://blog.pinaxproject.com/2016/11/22/how-configure-password-expiration-for-your-site/
Thanks
Ok, my problem was that I trusted the output of $ python manage.py makemigrations
that stated that No changes detected. However, when I did run $ python manage.py migrate, it applied a new migration for account_passwordhistory and account_passwordexpire tables just fine:
Applying account.0003_passwordexpiry_passwordhistory... OK
And the tables are in the DB.
Setting my password to expire in 1 second (python manage.py user_password_expiry my_user_name --expire 1) however did not have any effect on the site functionality, but that's a different issue.
I am practicing unit testing using django
In items/tests.py
class NewBookSaleTest(SetUpLogInMixin):
def test_client_post_books(self):
send_post_data_post = self.client.post(
'/booksale/',
data = {
'title':'Book_A',
}
)
new_post = ItemPost.objects.first()
self.assertEqual(new_post.title, 'Book_A')
In views/booksale.py
class BookSale(LoginRequiredMixin, View):
login_url = '/login/'
def get(self, request):
[...]
def post(self, request):
title = request.POST.get('title')
saler = request.user
created_bookpost = ItemPost.objects.create(
user=saler,
title=title,
)
# redis + celery task queue
auto_indexing = UpdateIndexTask()
auto_indexing.delay()
return redirect(
[...]
)
when I run unit test, raise redis connection error
redis.exceptions.ConnectionError
I know when I running redis-server and celery is error will solve
but when I run unit test in Travis CI I can't run redis-server and celery in Travis CI
So, I found this link
I try insert this code in .travis.yml
language:
python
python:
- 3.5.1
addons:
postgresql:"9.5.1"
install:
- pip install -r requirement/development.txt
service:
- redis-server
# # command to run tests
script:
- pep8
- python wef/manage.py makemigrations users items
- python wef/manage.py migrate
- python wef/manage.py collectstatic --settings=wef.settings.development --noinput
- python wef/manage.py test users items --settings=wef.settings.development
but it shows same error
so I found next link
before_script:
- sudo redis-server /etc/redis/redis.conf --port 6379 --requirepass 'secret'
but... it show same error...
how can I running redis-server in travis ci?
If you have not solve the problem now, here is a solution.
Remove the service line.
Redis is provided by the test environment as a default component, so
service:
- redis-server
will be translated as:
service redis start
In this problem, we want to customize redis to add password auth. So we doesn't need travis ci to start redis service. Just use the before_script.
And after all, your .travis.yml should be this:
language:
python
python:
- 3.5.1
addons:
postgresql:"9.5.1"
install:
- pip install -r requirement/development.txt
before_script:
- sudo redis-server /etc/redis/redis.conf --port 6379 --requirepass 'secret'
# # command to run tests
script:
- pep8
- python wef/manage.py makemigrations users items
- python wef/manage.py migrate
- python wef/manage.py collectstatic --settings=wef.settings.development --noinput
- python wef/manage.py test users items --settings=wef.settings.development
In one of my test files I call a Django management command:
def setUpModule():
management.call_command('loaddata', 'frontend/fixtures/chemicals.json',
verbosity=0)
management.call_command('create_indexes_and_matviews',
db_name, db_user, db_pass,
verbosity=2)
This test runs fine when I run it locally with manage.py test.
However, on Travis I get this error:
======================================================================
ERROR: setUpModule (frontend.tests.test_api_views)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/build/.../frontend/tests/test_api_views.py", line 35, in setUpModule
verbosity=2)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/django/core/management/__init__.py", line 95, in call_command
raise CommandError("Unknown command: %r" % name)
CommandError: Unknown command: 'create_indexes_and_matviews'
How can I let Travis know about the command?
This is my Travis file:
language: python
python:
- "2.7"
addons:
postgresql: "9.3"
env:
- SECRET_KEY=test DB_NAME=dbtest DB_USER=test DB_PASS=test
before_install:
- export DJANGO_SETTINGS_MODULE=....settings.local
- export PYTHONPATH=$HOME/builds/...
install:
- pip install -r requirements.txt
- pip install -r requirements/local.txt
before_script:
- psql -U postgres -c 'CREATE DATABASE dbtest;'
- psql -U postgres -c "CREATE EXTENSION postgis" -d dbtest
- psql -U postgres -c "CREATE EXTENSION postgis_topology" -d dbtest
- psql -U postgres -c "CREATE USER test WITH CREATEUSER PASSWORD 'test';"
- psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE dbtest to test;"
- psql -U postgres -c "ALTER USER test CREATEDB;"
- cd frontend && python manage.py migrate
script:
- python manage.py test
Is there something I should add so that it knows where to find management commands?
From my practice I know two reasons for such problem.
A. No the_app with create_indexes_and_matviews listed in settings.INSTALLED_APPS (it could be missed, excluded in if/else or try/except magic)
To check actual settings, try to add following command to the Travis file
echo "from django.conf import settings;print(settings.INSTALLED_APPS)" | python manage.py shell
B. Missed app dependencies. Try to get the actual error on travis with following command:
echo "from the_app.management.commands.create_indexes_and_matviews import Command" | python manage.py shell
Usually, real import error is descriptive enough to find the fix.
Does anyone have an example of successfully using flask-migrate on pythonanywhere? Does anyone have a simple example of what an app.py using migration should look like? Something along the lines of:
from flask import Flask, request, render_template
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.migrate import Migrate, MigrateCommand
from flask.ext.script import Manager
app = Flask(__name__ )
app.secret_key = 'This is really unique and secret'
app.config['SQLALCHEMY_DATABASE_URI'] = '<whatever>'
db = SQLAlchemy(app)
db.create_all()
manager = Manager( app )
migrate = Migrate( app, db )
manager.add_command('db', MigrateCommand )
I find that when I try to run
python app.py db init
it fails to generate the migration repository. Could this be a file permission issue on pythonanywhere?
Author of Flask-Migrate here.
The idea is that you generate your db repository in your development environment. You have to commit your repository along with your source files to source control.
Then when you install the application on your hosting service all you need to do is run the upgrade command to get your db created and migrated to the latest revision.
Update: base on your comments below, you want to develop an application from scratch. I just tested this myself, and was able to create a db repository, create a migration, and apply it. What I did is start the pythonanywhere bash console. Here is a copy of my complete session:
17:39 ~ $ mkdir dbtest
17:39 ~ $ cd dbtest
17:39 ~/dbtest $ virtualenv venv
New python executable in venv/bin/python2.7
Also creating executable in venv/bin/python
Installing setuptools............done.
Installing pip...............done.
17:39 ~/dbtest $ . venv/bin/activate
(venv)17:39 ~/dbtest $ pip install flask flask-migrate
...
(venv)17:42 ~/dbtest $ vi dbtest.py
... (entered flask-migrate example code, see below)
(venv)17:47 ~/dbtest $ python dbtest.py
usage: dbtest.py [-?] {shell,db,runserver} ...
positional arguments:
{shell,db,runserver}
shell Runs a Python shell inside Flask application context.
db Perform database migrations
runserver Runs the Flask development server i.e. app.run()
optional arguments:
-?, --help show this help message and exit
(venv)17:47 ~/dbtest $ python dbtest.py db init
Creating directory /home/miguelgrinberg/dbtest/migrations ... done
Creating directory /home/miguelgrinberg/dbtest/migrations/versions ... done
Generating /home/miguelgrinberg/dbtest/migrations/README ... done
Generating /home/miguelgrinberg/dbtest/migrations/alembic.ini ... done
Generating /home/miguelgrinberg/dbtest/migrations/env.py ... done
Generating /home/miguelgrinberg/dbtest/migrations/script.py.mako ... done
Generating /home/miguelgrinberg/dbtest/migrations/env.pyc ... done
Please edit configuration/connection/logging settings in '/home/miguelgrinberg/dbtest/migrations/alembic.ini' before proceeding.
(venv)17:54 ~/dbtest $ python dbtest.py db migrate
INFO [alembic.migration] Context impl SQLiteImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected added table 'user'
Generating /home/miguelgrinberg/dbtest/migrations/versions/1c4aa671e23a_.py ... done
(venv)17:54 ~/dbtest $ python dbtest.py db upgrade
INFO [alembic.migration] Context impl SQLiteImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
INFO [alembic.migration] Running upgrade None -> 1c4aa671e23a, empty message
(venv)17:55 ~/dbtest $ ls -l
total 8
-rw-r--r-- 1 miguelgrinberg registered_users 3072 Sep 28 2014 app.db
-rwxrwxr-x 1 miguelgrinberg registered_users 511 Sep 28 17:48 dbtest.py
drwxrwxr-x 3 miguelgrinberg registered_users 100 Sep 28 17:55 migrations
drwxrwxr-x 6 miguelgrinberg registered_users 52 Sep 28 17:41 venv
The example application that I used to test this is below:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.script import Manager
from flask.ext.migrate import Migrate, MigrateCommand
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///app.db'
db = SQLAlchemy(app)
migrate = Migrate(app, db)
manager = Manager(app)
manager.add_command('db', MigrateCommand)
class User(db.Model):
id = db.Column(db.Integer, primary_key = True)
name = db.Column(db.String(128))
if __name__ == '__main__':
manager.run()
Any chance you already have a migration repository created? Or a database?