This question is about the usage pattern of flask-migrate when it comes time to deploy. To set up a server or a docker container with your application, you need to create the databases.
Typically as in https://github.com/miguelgrinberg/flasky, the migrations folder is in the root of the project. This makes sense, but it means that in production, the migrations folder is not available if you are pulling the flask application as an installed package.
Is the correct pattern to copy the migrations folder to the container and run an upgrade there, or something else entirely? This seems awkward, because I would have to keep migrations in sync with the version of the app that I'm pulling from the python package repo. I am aware that it is possible to forego migrations entirely and just do db.create_all(), but if that is the answer, then I may be confused about the purpose of db migrations.
You can include files into a package with two-step:
1.set include_package_data to True in setup.py:
from setuptools import find_packages, setup
setup(
name='myapp',
version='1.0.0',
packages=find_packages(),
include_package_data=True, # <--
zip_safe=False,
install_requires=[
'flask',
],
)
2.Include the file pattern in MANIFEST.in:
graft myapp/static
graft myapp/templates
graft migrations # <--
This files will be included when you build the package. See here for the full MANIFEST.in command available.
Related
I am writing a reusable Django app and having problems creating migrations.
I have looked at this question, and I'm still confused. I have the following sort of directory structure:
django-mycleverapp/
django-mycleverapp/django_mycleverapp/
django-mycleverapp/django_mycleverapp/__init__.py
django-mycleverapp/django_mycleverapp/apps.py
django-mycleverapp/django_mycleverapp/models.py
django-mycleverapp/django_mycleverapp/urls.py
django-mycleverapp/django_mycleverapp/views.py
django-mycleverapp/example/
django-mycleverapp/example/manage.py
django-mycleverapp/example/example/
django-mycleverapp/example/example/__init__.py
django-mycleverapp/example/example/settings.py
django-mycleverapp/example/example/urls.py
django-mycleverapp/setup.py
As you can see, the directory "django_mycleverapp" contains my reusable app and the directory "example" contains a test project.
I include the models of "django_mycleverapp" in the INSTALLED_APPS section of the settings for "example".
However, running python ~/example/manage.py makemigrations django_mycleverapp doesn't build any migrations.
Any suggestions?
How am I meant to have a test project that will build migrations in "/django-mycleverapp/django_mycleverapp/migrations"?
Your app should be in the directory of your project. Your directory hierarchy should look like this.
django-mycleverapp/
django-mycleverapp/example/
django-mycleverapp/example/django_mycleverapp/
django-mycleverapp/example/django_mycleverapp/__init__.py
django-mycleverapp/example/django_mycleverapp/apps.py
django-mycleverapp/example/django_mycleverapp/models.py
django-mycleverapp/example/django_mycleverapp/urls.py
django-mycleverapp/example/django_mycleverapp/views.py
django-mycleverapp/example/manage.py
django-mycleverapp/example/example/
django-mycleverapp/example/example/__init__.py
django-mycleverapp/example/example/settings.py
django-mycleverapp/example/example/urls.py
django-mycleverapp/example/setup.py
If you do not want your app to be part of your "example" project, but rather want it to be separated and used by your project "example", you'll have to install it in your project using pip (this requires to have a setup.py at the root of your app).
For instance if you have published your app on a git repository "https://gitlab.com/myuser/myproject.git", you can add to our requirements.txt:
git+https://gitlab.com/myuser/myproject.git#v1.0#egg=myapp_name
If you don't have your app published on a git repository yet, you can add the absolute path to your app to you requirements.txt:
/path/to/django-mycleverapp/django_mycleverapp/
Don't forget to work in a virtualenv when you use pip.
I have an application design using flask with flask-sqlalchemy. In order to control database migrations as the models change I am using flask-migrate to wrap alembic and use within the flask-script management context.
I am trying to decide how to split the package distribution to achieve the following goals
Minimal set of dependencies for the main application package
Allow distribution of management scripts and test data for migrations and deployments perhaps using a secondary package depending on the main application module
The project structure is as follows
/
tests/ #test data
migrations/ #alembic root include env.py and alembic.ini
myapp/ # application package
setup.py
manage.py
wsgi.py
My manage.py looks like a lot the following. This is how I avoid any alembic or flask-migrate dependencies in the main application package, by attaching the Migrate object to the app in manage.py. This also allows me to control migration configurations in the same config file as the general flask/app configurations (as the config context is pushed by the manager and with Migrate.init_app)
from myapp import db, create_app
from myapp.database import database_manager #sub manager for creating/dropping db
from flask_migrate import Migrate, MigrateCommand
def _create_app(*args, **kwargs):
app = create_app(*args, **kwargs)
if migration is not None:
migration.init_app(app)
return app
manager = Manager(_create_app)
migration = Migrate(db = db)
manager.add_command('database', database_manager)
manager.add_command('migration', MigrateCommand)
if __name__ == "__main__":
manager.run()
The application submanager myapp.database.database_manager enables such commands as python manage.py database create/drop/test_data which uses sqlalchemy to create tables and populate the table with test_data from tests/ directory, but does not hook in any migration scripts, and allows me to use python manage.py migration init/revision/migrate/... to execute flask-migrate/alembic commands using the application configuration context.
I am trying to distribute this application to deploy on our internal servers. There are two distribution use cases as they are currently being used
Install new server
Install source distribution
Create new config file to reflect DB host etc.
Use manage.py -c /path/to/config to create database tables with database create
Point HTTP Server to wsgi.py
Update existing server
Update source distribution
Use manage.py -c /path/to/server/config migration upgrade to pull up the databases using the current app context
Server continues to point to wsgi.py
I would like to move this distribution to wheels/packages so that I can automate deployment on our corp intranet, So what I'm seeing here is that I would like to split my package into the main package myapp with only Flask framework dependencies and wsgi entrypoint and myapp-manage which includes the manage.py , tests, and migrationsdirectory.
So in the perfect world, the application will be installed on the system with global configuration files, and a specific management user will have access to the management package to perform upgrades.
Right now I am angling towards splitting the distribution with setup-app.py and setup-manage.py to create two seperate packages from the same source distribution. Is there a more appropriate way to package migrations and management scripts with a flask application? I have dug through the docs and while its clear how to package a flask application, distribution strategies for management scripts and migrations is less clear.
I'm using git to version control a Django 1.7 + Django CMS 3.0.6 project.
In the course of building various apps etc I'm ending up with a lot of migration files. The migration files are currently included in my git repo.
Thus far I have been trying to avoid including the virtual env files in my repo directly as it seems rather messy and redundant. Instead I have thus far been including a pip requirements file in the repo and using that to recreate the virtual env when needed.
However, I have recently discovered that choosing to include the migration files in the repo seems to require including all of the virtual env files in the repo as well. I say this because upon deploying my project to a production server and trying to run any of the db commands (syncdb, makemigrations or migrate) via python manage.py I get the error:
KeyError: u"Migration image_gallery.0001_initial dependencies reference nonexistent parent node (u'cms', u'0004_auto_20141108_1256')"
whereas such error does not occur on my local machine, even after deleting the database.
I tracked the source of this error down to the fact that the virtual env on my local machine has a reference to '0004_auto_20141108_1256' (inside the django-cms package - it appears some cms migration info is recorded directly inside the virtual env directory itself) while that of the production environment does not - as the production venv is create thorough a pip requirements file. Therefore, the two virtual envs do not exactly match, even though all third party libs are the same. Currently I am not including the venv in my git repo.
So as I see it I have two options:
1. include the virtual env in my git repo
2. drop the migration files from git
Which option is better and why - or is there a third even better way?
The downside to #1 is unnecessary bloat. The downside to option #2 is one loses the migration history, something one might potentially want to keep.
You never commit the virtual env, it defeats the purpose; you just add unnecessary content to git.
Instead, freeze the requirements and commit the file:
pip freeze > requirements.txt
Install the packages on the server:
pip install -r requirements.txt
The problem is in my django settings.py file:
MIGRATION_MODULES = {
'cms': 'cms.migrations_django',
'menus': 'menus.migrations_django',
'djangocms_file': 'djangocms_file.migrations_django',
...
}
I had to introduce the above to get django-cms 3.0.6 to work with django 1.7, a consequence of the fact that migrations in django 1.7 are no longer done with South, as django 1.7 now has it's own migration system, while cms 3.0.6. still expects migrations to be managed by South by default.
However, the effect of the above config is to store migrations in the above described paths which in my case pointed straight to the virtual env. Thus migration info was getting stored within the virtual env dir, leading to problems in deploying to production.
To fix this I modified my project directory structure to include a folder called "migrations":
myproject/manage.py
myproject/migrations/
myproject/myproject/
...
And modified the config to be:
MIGRATION_MODULES = {
'cms': 'migrations.cms.migrations_django',
'menus': 'migrations.menus.migrations_django',
'djangocms_file': 'migrations.djangocms_file.migrations_django',
...
}
This has the effect of now storing all migration files in the django project itself (and by extension the git repo). As migration info is no longer in the virtual env directory, there is no longer any reason to consider the rather unattractive possibility of including the virtual env in the repo.
I've tried to find a concrete answer for this but failed. I'm rather new to openshift, django and git so I hope it's not too trivial and not a duplicate.
The situation is as follows:
1. I'm developing a Django based web application and source control it using git on a private bitbucket repo. I have my regular django application source tree as needed.
2. I wish to deploy my app on openshift which comes with a different directory tree. I've been able to successfully run this quickstart:
git://github.com/rancavil/django-openshift-quickstart.git
Basically what I try to achieve is a wrapper directory (and git project) which is based on the repo specified in #2. Within this wrapper one of the subdirectories (wsgi/my_project) should be based on my private repo specified in #1.
I wish to be able to pull recent changes from the private repo and then push and deploy to openshift these changes.
Any help would be greatly appreciated!
I was also struggling with openshift structure. But there is no need for doing the staff like you have proposed.
You can have any structure of a project you want. Openshift will need just wsgi/application or wsgi.py as an entry point. I think that having one more file in your project is hardly a problem.
My current structure for a project (got it running recently) is following:
|- .openshift (directory for build and deployment hooks like syncdb or collectstatic commands)
|- custom_project (directory that is you django project)
\- templates
|- settings.py
|- urls.py
|- manage.py
|- requirements.txt
|- wsgi.py
There is really no need to have project under wsgi folder. However there are few more pitfalls to be aware of. Like creating static folder or setting .htaccess for static files.
Including my wsgi.py:
#!/usr/bin/python
import os
import logging
try:
virtenv = os.environ['OPENSHIFT_PYTHON_DIR'] + '/virtenv/'
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
execfile(virtualenv, dict(__file__=virtualenv))
except (IOError, KeyError):
pass
os.environ.setdefault("DJANGO_SETTINGS_MODULE", 'custom_project.settings')
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
I have used a lot of resources to make my project running on openshift so I'm also including them for other.
bitbucket and openshit repos: Can I use my existing git repo with openshift?
Slides with necesary steps: http://appsembler.com/blog/paas-bakeoff-comparing-stackato-openshift-dotcloud-and-heroku-for-django-hosting-and-deployment/
And the most important one, the really well written step-by-step tutorial: http://www.paascheatsheet.com/#Introduction
I'm trying to set up Pinax and I'm new to everything involved (Django, Pinax, webservers, etc). I'm following http://pinax.readthedocs.org/en/latest/gettingstarted.html
When I generate a project using:
(mysite-env)$ pinax-admin setup_project -b basic mysite
The directory structure I get is:
apps __init__.py manage.py settings.pyc urls.py
dev.db __init__.pyc requirements static urls.pyc
fixtures locale settings.py templates wsgi.py
Which as far as I can tell is missing the deployment folder (when you compare to the directory structure shown here : http://pinax.readthedocs.org/en/latest/starterprojects.html). It doesn't seem to be effecting anything yet, but it makes me nervous. What is going on and is the fact I'm missing the deployment folder going to cause problems in the future?
I'm running Ubuntu and using python 2.7. I had the same behaviour with Windows 7, python 2.6
Thanks!
The new Django versions have made the old pinax pretty much useless. Now Django supports project templates and Pinax is separated into several smaller projects regarding starter projects (such as pinax-project-account) and apps (such as django-user-account).
The current way to use pinax is to choose a starter project, and then running something like:
$ django-admin.py startproject --template=https://github.com/pinax/pinax-project-account/zipball/master <project_name>
and then install requirements:
$ pip install -r requirements.txt
This will create a new Django project using the starter-project as a template, which already includes a few apps (like django-user-account) and templates (with bootstrap!). The project is ready to run, and already includes a bunch of functionality (like user registration, login and management).
Also, Django has changed the project directory structure a bit, so now it doesn't really look like that anymore.