I have created a Django proyect with 20 sites (one different domain per site) for 20 different countries. The sites share everything: codebase, database, urls, templates, etc.
The only thing they don't share are small customizations (logo, background color of the CSS theme, language code, etc.) that I set in each of the site settings file (each site has one settings file, and all of these files import a global settings file with the common stuff). Right now, in order to run the sites in development mode I'll do:
django-admin.py runserver 8000 --settings=config.site_settings.site1
django-admin.py runserver 8001 --settings=config.site_settings.site2
...
django-admin.py runserver 8020 --settings=config.site_settings.site20
I have a couple of questions:
I've read that it is possible to create a virtual host for each site (domain) and pass it the site's settings.py file. However, I am afraid this would create one Django instance per site. Am I right?
Is there a more efficient way of doing the deployment? I've read about django-dynamicsites but I am not sure if it is the right way to go.
If I decide to deploy using Heroku, it seems that Heroku expects only one settings file per app, so I would need to have 20 apps. Is there a solution for that?
Thanks!
So, I recently did something similar, and found that the strategy below is the best option. I'm going to assume that you are familiar with git branching at this point, as well as Heroku remotes. If you aren't, you should read this first: https://devcenter.heroku.com/articles/git#multiple-remotes-and-environments
The main strategy I'm taking is to have a single codebase (a single Git repo) with:
A master branch that contains all your shared code: templates, views, URLs.
Many site branches, based on master, which contain all site-specific customizations: css, images, settings files (if they are vastly different).
The way this works is like so:
First, make sure you're on the master branch.
Second, create a new git branch for one of your domains, eg: git checkout -b somedomain.com.
Third, customize your somedomain.com branch so that it looks the way you want.
Next, deploy somedomain.com live to Heroku, by running heroku create somedomain.com --remote somedomain.com.
Now, push your somedomain.com branch code to your new Heroku application: git push somedomain.com somedomain.com:master. This will deploy your code on Heroku.
Now that you've got your somedomain.com branch deployed with its own Heroku application, you can do all normal Heroku stuff by adding --remote somedomain.com to your normal Heroku commands, eg:
heroku pg:info --remote somedomain.com
heroku addons:add memcache:5mb --remote somedomain.com
etc.
So, now you've basically got two branches: a master branch, and a somedomain.com branch.
Go back to your master branch, and make another new branch for your next domain: git checkout master; git checkout -b anotherdomain.com. Then customize it to your liking (css, site-specific stuff), and deploy the same way we did above.
Now I'm sure you can see where this is going by now. We've got one git branch for each of our custom domains, and each domain has it's own Heroku app. The benefit (obviously) is that each of these project customizations are based off the master branch, which means that you can easily make updates to all sites at once.
Let's say you update one of your views in your master branch--how can you deploy it to all your custom sites at once? Easily!
Just run:
git checkout somedomain.com
git merge master
git push somedomain.com somedomain.com:master # deploy the changes
And repeat for each of your domains. In my environment, I wrote a script that does this, but it's easy enough to do manually if you'd like.
Anyhow, hopefully that helps.
Related
I am developing Django Wagtail application on my local machine connected to a local postgres server.
I have a test server and a production server.
However when I develop locally and try upload it there is always some issue with makemigration and migrate e.g. KeyError etc.
What are the best practices of ensuring I do not get run into these issues? What files do I need to port across?
so ill tell you what i do and what most of the companies that i worked as a django developer did and i can tell you by experience that worked pretty well.
First containerize your application, this will make your life much more easy and you will remove external influence in your code, also will get you an easy way to reproduce your environment.
Your Dockerfile should be from some python image and should do 3 basically things:
Install your requirements dependencies
Run the python manage.py migrate --noinput command
Run a http server such as gunicorn with gunicorn -c /gunicorn.py wsgi:application
You ill do the makemigration in your local machine and make sure that everything is working before commit then to the repo.
In your gunicorn.py you ill put your settings to run the app such as the number of CPU, the binding port, the folder that your app is, something like:
import os
import multiprocessing
# Chdir to specified directory before apps loading.
# https://docs.gunicorn.org/en/stable/settings.html#chdir
chdir = '/app/'
# Bind the application on localhost both on ipv6 and ipv4 interfaces.
# https://docs.gunicorn.org/en/stable/settings.html#bind
bind = '0.0.0.0:8000'
Second containerize your other stuff, for example the postgres database, the redis (for cache), a connection pooler for the database depending on the size of your application.
Its highly recommend that you have a step in the pipeline to do tests, they need to run before everything, maybe just after lint
Ok what now? now you need a way to deploy that stuff, the best for that scenario is: pull your image to github registry, and you can add a tag to that for example:
IMAGE_ID=ghcr.io/${{ github.repository_owner }}/$IMAGE_NAME
# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
docker tag $IMAGE_NAME $IMAGE_ID:staging
docker push $IMAGE_ID:staging
This can be add in a github action in the build step for example.
After having your new code in a new image inside github you just need to update the current one, this can be done by creaaing a script to do it in the server and running that script from github action, is something like:
docker pull ghcr.io/${{ github.repository_owner }}/$IMAGE_NAME
echo 'Restarting Application...'
docker stop {YOUR_CONTAINER} && docker up -d
sudo systemctl restart nginx
echo 'Cleaning old images'
sudo docker system prune -af
You can see that i create the image with a staging tag, you can create a rule in github actions for example to trigger that action when you create a new release for example, and create another action to be trigger in every new commit and build/deploy for a dev tag.
For the migration problem, the first thing is, when your application go live squash every migration to the first one (you can drop the database and all the migration then create the database and run the makemigration command again to reach this), so you can have a clean migration in the server. Never creates unnecessary relation between the tables, prefer always doing cached properties instead of add new columns, use UUID for unique ids, and try to not do breaking changes in the database, its hard but if you plan the database before is not so difficult to do.
Hit me if you have any questions. A lot of the stuff that i said can be done in a lot of other platforms such as gitlab, travis, circle ci, but i use the github action in the example because i think is more simple to picture.
EDIT:
I forgot to tell you to have a cron in your server doing backups of your databases, the migrate command ill apply the changes only after the verification but if something else break the database this can save your life.
I have a git repository with two Django 1.5 projects: one for a website, the other for a REST api. The git repository looks like this:
api_project/
www_project/
logs/
manage.py
my_app_1/
my_app_2/
The manage.py file defaults to www_project.settings. To launch the api_project, I run:
DJANGO_SETTINGS_MODULE=api_project.settings ./manage.py shell
I guess I could setup 3 git repositories, one with the common apps, one for the api project and one for the www project, using git submodules and all, but it really seems overkill. Up to now, everything worked fine.
But now I'm trying to deploy this setup using Chef. I'd like to use the application and application_python cookbooks, and run my django projects with gunicorn, but these cookbooks seem to be meant to deploy only one project at a time.
Here's what my chef recipe looks like for the www_project:
application "django_app" do
path "/var/django"
owner "django"
group "django"
repository "git.example.com:blabla"
revision "master"
migrate true
packages ["libevent-dev", "libpq5" , "git"]
# libevent-dev for gevent (for gunicorn), libpq5 for postgresql
environment "DJANGO_SETTINGS_MODULE" => "www_project.settings"
# for syncdb and migrate
django do
local_settings_file "www_project/settings.py"
settings_template "settings.py.erb"
purge_before_symlink ["logs"]
symlinks "logs" => "logs"
collectstatic true
database do
database "blabla"
engine "postgresql_psycopg2"
username "django"
password "super!password"
end
database_master_role "blabla_postgresql_master"
migration_command "/var/django/shared/env/bin/python manage.py" +
" syncdb --noinput && /var/django/shared/env/bin/python" +
" manage.py migrate"
end
gunicorn do
app_module "www_project.wsgi:application"
preload_app true
worker_class "egg:gunicorn#gevent"
workers node['cpu']['total'].to_i * 2 + 1
port 8081
proc_name "blabla_www"
end
end
I would just like to know how to add another gunicorn ressource for the api_project.
Has anyone run into a similar problem? Would you recommend patching my local copy of the application_python cookbook to allow multiple projects in one git repo? Or should I go through the pain of setting up 3 separate git repositories? Or any other solution?
Thanks!
You can separate your code into two separate "application" blocks, as all the resources defined inside are sub-resources and the actual execution is done at the level of "application".
Another solution would be to fork/patch the application_python providers django and gunicorn to allow more complex behaviors, for example allowing more than one application to be deployed. Although it is probably not required by so many users to merit all the effort and complexity.
Until now I used svn as source control. At this time I have started a new project and it is stored on gitHub.
Issue is that Heroku and GitHub, both use git. First one to publish app and second one for version control.
My app schema is:
base-dir <--github base
some-text-files (Readme, ... )
django-project-dir <--heroku base
manage.py
main-app-dir
settings.py
other-app-dirs
views.py
models.py
When I push to gitHub base-dir and all subfolders are pushed.
To Heroku only django-project-dir should be pushed.
Notice: I have tried to create a new git repository at django-project-dir level but git take it as a submodule and excluded from gitHub.
Because this is a new project I can easily change to another schema dirs.
My question:
What is the easy way to coexist both Heroku and GitHub git configurations?
Your best option is probably to push the full repository to Heroku, but make sure Heroku ignores all files not required to run your application (see https://devcenter.heroku.com/articles/slug-compiler). Alternatively, consider creating two repositories (one for documentation and one for production code).
Your best bet is to move you readme and other files to your project root. Then just add GitHub as a separate remote (when you're in your project directory).
git remote add origin https://github.com/USERNAME/REPO
Then you can push to GitHub with git push origin master. You will have to do a forced push (the -f option) the first time assuming you're pushing what used to be the repo you used exclusively for Heroku.
You'll still be able to push to Heroku with git push heroku master.
You should have two remotes.
This is good and even desirable.
You have github and that's your remote code repository of record.
Then you have a current deployment via heroku and that is the 2nd remote.
Heroku is actually set up to use git as part of the system of pushing changes to your site on it.
My problem/issue
We are working on an opensource project which we have hosted on github. The project is written in Django and so we have our settings in a settings.py file. We have it open source not because we are too cheap for a subscription, but because we intentionally want it to be open source.
Anyways, we also want to run this code ourselves and so we have a site_settings.py which contains all sensitive data like the db password etc. Since we don't want this on github for everyone to see, this is in the .gitignore file.
Normally we clone the repo and add this file locally and on the server, this way we all have our personal settings and sensitive data isn't stored on git.
The problem is that we now want to run our code on Heroku, which doesn't support the creation of files on the server, since heroku doesn't work like a normal server ( it works with a distributed file system).
Of course I have thought about it myself, but I am not sure if this is the right approach. Please keep in mind that I am both new to Heroku and Git. The following is my current solution:
My current idea on how to fix this
There are two remotes:
origin == Github
heroku == Heroku
leaving out all the development and feature branches, we only have both the master branches (on Heroku and Github) to worry about.
My idea was to have the branches set up like this locally:
master -> remotes/origin/master
heroku -> remotes/heroku/master
On the master branch we put the site_settings.py in the .gitignore so git will ignore it when committing to Github, the same goes for all the dev branches etc. We collect all our changes in the master branch until we have a new release.
On the heroku branch we edited the .gitignore file to stop ignoring the site_settings.py, so it will be committed to Heroku once we push to the heroku branch.
Once we have a new release we switch to heroku branch and merge master into heroku like so:
git checkout heroku
git pull
git merge master
Once we have sorted out merge conflicts from the merge, we commit to Heroku and we have a new release up and running.. =D
My question
Is my idea acceptable to solve this (will it even work with the .gitignore like this?) and if not how do we solve this in the cleanest/best way possible? If you need more info don't hesitate to ask :)
For starters, storing your config in files is the wrong way to go about this, so don't do it.
From the 12factor app:
An app’s config is everything that is likely to vary between deploys
(staging, production, developer environments, etc). This includes:
Resource handles to the database, Memcached, and other backing
services
Credentials to external services such as Amazon S3 or Twitter
Per-deploy values such as the canonical hostname for the deploy
Apps
sometimes store config as constants in the code. This is a violation
of twelve-factor, which requires strict separation of config from
code. Config varies substantially across deploys, code does not.
A litmus test for whether an app has all config correctly factored out
of the code is whether the codebase could be made open source at any
moment, without compromising any credentials.
Note that this definition of “config” does not include internal
application config, such as config/routes.rb in Rails, or how code
modules are connected in Spring. This type of config does not vary
between deploys, and so is best done in the code.
Another approach to config is the use of config files which are not
checked into revision control, such as config/database.yml in Rails.
This is a huge improvement over using constants which are checked into
the code repo, but still has weaknesses: it’s easy to mistakenly check
in a config file to the repo; there is a tendency for config files to
be scattered about in different places and different formats, making
it hard to see and manage all the config in one place. Further, these
formats tend to be language- or framework-specific.
The twelve-factor app stores config in environment variables (often
shortened to env vars or env). Env vars are easy to change between
deploys without changing any code; unlike config files, there is
little chance of them being checked into the code repo accidentally;
and unlike custom config files, or other config mechanisms such as
Java System Properties, they are a language- and OS-agnostic standard.
Another aspect of config management is grouping. Sometimes apps batch
config into named groups (often called “environments”) named after
specific deploys, such as the development, test, and production
environments in Rails. This method does not scale cleanly: as more
deploys of the app are created, new environment names are necessary,
such as staging or qa. As the project grows further, developers may
add their own special environments like joes-staging, resulting in a
combinatorial explosion of config which makes managing deploys of the
app very brittle.
In a twelve-factor app, env vars are granular controls, each fully
orthogonal to other env vars. They are never grouped together as
“environments,” but instead are independently managed for each deploy.
This is a model that scales up smoothly as the app naturally expands
into more deploys over its lifetime.
For Python, your config can be found in os.environ. For specific config, particularly the database you want to be using something like:
import os
import sys
import urlparse
# Register database schemes in URLs.
urlparse.uses_netloc.append('postgres')
urlparse.uses_netloc.append('mysql')
try:
# Check to make sure DATABASES is set in settings.py file.
# If not default to {}
if 'DATABASES' not in locals():
DATABASES = {}
if 'DATABASE_URL' in os.environ:
url = urlparse.urlparse(os.environ['DATABASE_URL'])
# Ensure default database exists.
DATABASES['default'] = DATABASES.get('default', {})
# Update with environment configuration.
DATABASES['default'].update({
'NAME': url.path[1:],
'USER': url.username,
'PASSWORD': url.password,
'HOST': url.hostname,
'PORT': url.port,
})
if url.scheme == 'postgres':
DATABASES['default']['ENGINE'] = 'django.db.backends.postgresql_psycopg2'
if url.scheme == 'mysql':
DATABASES['default']['ENGINE'] = 'django.db.backends.mysql'
except Exception:
print 'Unexpected error:', sys.exc_info()
For more info on config vars, see here: https://devcenter.heroku.com/articles/config-vars
Rather than developing a complex git-based deployment of settings files and variables, on Heroku, I'd use the environment variables for all of your sensitive stuff. That way you don't have to keep them in code. See database docs and environment variables docs for information - put all this stuff in there, and then access via os.getenv() python docs.
To see your environment variables for the app, using the heroku cli tool:
heroku run env
Will print out all the variables. To add new variables (e.g. S3 keys etc) use:
heroku config:add VAR_NAME=value
Here is my current setup:
GitHub repository, a branch for dev.
myappdev.appspot.com (not real url)
myapp.appspot.com (not real url)
App written on GAE Python 2.7, using django-nonrel
Development is performed on a local dev server. When I'm ready to release to dev, I increment the version, commit, and run "manage.py upload" to the myappdev.appspot.com
Once testing is satisfactory, I merge the changes from dev to main repo. I then run "manage.py upload" to upload the main repo code to the myapp.appspot.com domain.
Is this setup good? Here are a few issues I've run into.
1) I'm new to git, so sometimes I forget to add files, and the commit doesn't notify me. So I deploy code to dev that works, but does not match what is in the dev branch. (This is bad practice).
2) The datastore file in the git repo causes issues. Merging binary files? Is it ok to migrate this file between local machines, or will it get messed up?
3) Should I be using "manage.py upload" for each release to the dev or prod environment, or is there a better way to do this? Heroku looks like it can pull right from GitHub. The way I'm doing it now seems like there is too much room for human error.
Any overall suggestions on how to improve my setup?
Thanks!
I'm on a pretty similar setup, though I'm still runing on py2.5, django-nonrel.
1) I usually use 'git status' or 'git gui' to see if I forgot to check in files.
2) I personally don't check in my datastore. Are you familiar with .gitignore? It's a text file in which you list files for git to ignore when you run 'git status' and other functions. I put in .gaedata as well as .pyc and backup files.
To manage the database I use "python manage.py dumpdata > file" which dumps the database to a json encoded file. Then I can reload it using "python manage.py loaddata".
3) I don't know of any deploy from git. You can probably write a little python script to check whether git is up to date before you deploy. Personally though, I deploy stuff to test to make sure it's working, before I check it in.