Good way to deploy django project to a testing sever? - django

This is specific to my current project. But maybe the answers will reveal some more generic solutions.
Here is the situation:
I develop django project on my Windows box
I use SVN to commit to a SVN repository
while developing I use development server that comes with django
There is a testing server (apache) that runs somewhere else, and everytime i finish something I need to manually copy my work via WinSCP/Putty and make sure it works on testing server
Testing server is accessible for our testers to use and test and report bugs
I would like to automate this process as it is very painful. It involves me to export the whole repository, copy to the testing server, get rid of the pyc files, sometimes restarting apache, use the correct settings.py (usually some renaming).
I would like to for the testing server to automatically retrieve new files after each SVN commit. I could write a custom script to do all this stuff, but something tells me that there are some easy-to-use solutions I could use to change my workflow to make things less painful.
One extra bonus. There is a designer that works on HTML/CSS on the templates directly on the testing server. I need to check whether he made changes and I transfer them to my computer and subsequently to SVN rep. My boss thinks it's too dangerous to give him SVN access. Any ideas to help me out with this, also?

deployment:
I would say its better to do the deployment the same way you do for the production. Fabric is a good solution.
SVN way:
If you want to do it in the SVN way, create a branch called testing, once you have a working version of your code and ready for testing merge the development branch to the testing branch. Make sure you have permission on the testing branch to restrict everyone from merging to the test branch. After merging the test team should take a update to a specific version.
.pyc
It unnecessary to manually remove pyc files you can add a svn hook which can ignore the pyc files on commit. Create a file .svnignore
*.pyc
and run this command
svn -R propset svn:ignore -F .svnignore .
If you’ve already got yourself into the mess of versioning your compiled files, then you can do either of these things.
find -name "*.pyc" -exec svn revert {} \;
find -name "*.pyc" -exec svn delete {} \;
Django settings file
You can set the environment variable through which django can take up corresponding settings file. Django set env
Designer
well designer working directly on the test server is not a bonus point. :) its a headache. In a ideal environment no one should touch the code in the testing server. Create a separate branch for the designer or he can commit to the dev branch which all the developers can merge.

One option is to create a read only svn user and have it checkout the svn repository on the apache server. Then to do a build you run 'svn update'. You can check if the designer modified files by doing a 'svn status'.
If your svn repository is on the same machine as your qa django instance you could use a post-commit hook to have it svn update after each commit, and bounce apache if needed. See http://subversion.tigris.org/faq.html#website-auto-update

Related

Git - How to commit a local repository to a subfolder of another local repository?

I have a Django project that I've started some time ago and I was hosting it at bitbucket. Now I need to host it at openshift, and the way to do that is that they provide you with a git repository and every time you push they deploy automatically. The problem is that the repository comes with several top-level folder for configuration and setup, and the effective django project must be inside a subfolder called wsig/openshift.
My question is, how can I commit my changes from my local django repository to the wsig/openshift subfolder of my local openshift repository? Because I intend to continue to develop on the bitbucket/local repository
You are probably looking for submodules. From the docs:
Submodules allow foreign repositories to be embedded within a
dedicated subdirectory of the source tree, always pointed at a
particular commit.
So you would do and would have the bitbucket repository as a seperate repository embedded in a subfolder of the openshift repository by running
git submodule add path_to_bitbucket folder/in/openshift
in the openshift repository.
You will have to run an occaisonal git submodule update to keep openshift up to date but you probably already expected extra work of that sort.
I had the exact same problem too! Is highly annoying but I took another road:
Why don't you create a Python 2.7 project from scratch? Current Django structure is honestly annoying. The way I did was:
Create an openshift project which was in that annoying structure.
Copy, preserve (in my local FS, not in Openshift) a copy of settings.py and wsgi.py.
Discard that project, and start a bare Python 2.7 project.
Check it out to my local FS, create a Django project in my local fs.
Replace the contents of wsgi and settings accordingly (adapting any possible misplaced paths - it's easier than it looks).
Commit/push (this new structure).
You will do differently in point 4: you will checkout that remote branch (bitbucket) as well, merge it on the openshift branch, change accordingly those files in point 5, and push the openshift branch.
There you have a brand-new project, matching your structure (perhaps you want to configure both remote branches in your environment: openshift and bitbucket).
That's the way I did and honestly I have nothing to regret.
Offtopic, but perhaps would be useful since you're using Django: This is specially important if you want to -also- use (gunicorn|uwsgi)+nginx (with a custom cart. which does not provide apache but nginx, and python), and so cannot use the default Django cart.

Pydev + Django workflow. Local(test) + remote synchronization. Using git with django

I'm new to django and my very first project is my blog. I wonder how django developers who use pydev normally synchronize with remote hosting server, updating their sites?
I also would like to know, how do you combine usage of git with a django project? Should I just make a repository for the entire project?
At my company we've got an entire git repository for each project, including the Django sources that are put in the PYTHONPATH for each project, making Django versions project dependant. The folder structure is something like:
/.git
/projectname/app1
/projectname/app2
/projectname/manage.py
/django-lib/django/...
As django-lib is not a Python module, we include both / and /django-lib in the PYTHONPATH. If your project is becoming large, you might want to consider using git submodules on your apps.
We've also setup several servers to support the developers. There's a testing server running a central testing database and a setup including Apache with WSGI to make testing on a real server possible, which sometimes is a bit different then the local manage.py the developers use before committing their changes.
The testing server is updated with the master branch of our git repository. We've made several scripts to allow all developers to do this without letting them login to the server via SSH, but that is just during pre-release. After release, that server will become our staging server, and we'll remove all scripts from it to make it just like our production server.
Every developer has setup their local project to make sure that it communicates with the central testing database, containing several test data. I myself push my changes from the commandline, but you could also use EGit for this.
When we've got a release, we put it in a separate branch, called 'release' (obviously) and the production server will pull only from that branch. This is done via SSH, but I don't really know how your server setup looks like, so I guess that that last step is entirely up to you.
I hope that this has helped you a bit. I won't say that this is the best workflow possible, but it works for us and you should figure out what works for you.
Most experienced Django developers use pip(or distribute) and virtualenv deal with all the python packages you might need for your Django projects (including Django itself).
Personally, all I keep in my projects git repository is a bunch of segregated requirements lists generated by pip :
. ~/Dev/environs/$PROJECT_NAME/bin/activate
pip freeze > ./docs/requirements/main.list
I'm fairly sure most django developers would be familiar with Fabric, which I use for :
streamlining local interaction with git and,
pushing to our central repository,
pulling from our production or test server
touching the wsgi on the relevant server
and pretty much any other kind of task you might find yourself using ssh terminal session for.
For those cases where I need to make changes to someone elses django application in order to make it work or suit our purposes, I :
fork it on github,
clone from my forked repo
make the changes
push it up to my own repo
and provide merge requests to the original repo owner
This way, i have a repo where i can use pip requirement lists to keep pulling from until the original application owner gets their own repo updated.

Django + SVN + Deployment

I'm a strong proponent of version control, and am starting work on a Django project. I've done a few before, and have tried a few different approaches, but I haven't yet found a decent structure that I actually feel comfortable with.
Here's what I want:
a) Source code checked into version control
b) Preferably the environment is not checked into version control (something like buildout or pip requirements.txt is fine for setting up the environment)
c) A reasonable "get a new developer going" story
d) A reasonable deployment story - preferably the entire deployment environment could be generated by a script on the server
It seems to me like someone has to have done this before, but many hours of searching have all led to half-baked solutions that don't really address all of these.
Any thoughts on where I should look?
Look at fabric to manage deployments.
This is what I use to manage servers/deployments with fabric: louis (it is just a collection of fabric commands). I keep a louisconf.py file with each project.
I'd recommend using a distributed VCS (git, hg,...) instead of svn. The reason being that the ease of branching allows for several schemes for deployment. You can have, for example, production and staging branches. Then you enforce that the only merges into production happen from staging by convention.
As for getting developers started quickly you have it right with pip and requirements.txt. I think that also means that you are using virtualenv, but if not that's the third piece. I'd recommend getting a basic README in place. Have the first assignment of each developer that joins a project be to update the README.
The rough way to get someone on board is to have her checkout the code, create a virtualenv, and install the requirements.
I'd recommend having a settings.py file that works with sqlite3 and such that a new developer can use to just get going fast (ie after installing the requirements). However, how you manage the different settings files depends on your project layout. There should be some set of default settings for new developers to use, though.
I keep a projects/ directory in my home directory (on Linux). When I need to start a new project, I make a new, shortly-named (that sufficiently describes the project) dir in projects/; that becomes the root of a new virtualenv (with --no-site-packages) for that project.
Inside that dir (after I've installed the venv, sourced it, and installed the copy of django I'll be working with), I "django-admin.py startproject" a subdir, normally by the same short name. That dir becomes the root of my hg repo (with a quick hg init and ci), no matter how small the project.
If there's any chance of sharing the project with other developers (a project for work, for example), I include a pip requirements.txt at the repo root. Only project requirements go in there; django-debug-toolbar and django-extensions, staples for my dev workflow, are not project requirements, for example. South, when we use it, is.
As for the django project, I normally keep the default settings.py, possibly with a few changes, and add the local_settings convention to the end of it (try: from local_settings import *; except ImportError: pass). My and other devs' specific environment settings (adding django-extensions and django-debug-toolbar to installed apps, for example) go in local_settings.py, which is not checked in to version control. To help a new dev out, you could provide a template of that file as local_settings.py.temp, or some other name that won't be used for any other purpose, but I find that this unnecessarily clutters the repo.
For personal projects, I normally include a README if I plan on releasing it publicly. At work, we maintain Trac environments and good communication to get new devs up to speed on a project.
As for deployment, as rz mentioned, I hear fabric is really good for that kind of automated local/remote scripting, though I haven't really taken the chance myself to look into it.
For the uninitiated, a typical shell session for this might look like the following:
$ cd ~/projects/
$ mkdir newproj
$ cd newproj/
$ virtualenv --no-site-packages .
$ source bin/activate
(newproj)$ pip install django django-debug-toolbar django-extensions
... installing stuff ...
(newproj)$ django-admin.py startproject newproj
(newproj)$ cd newproj/
(newproj)$ hg init .; hg ci -A -m "Initial code"

Django Deployment Advice

I have a multi-step deployment system setup, where I develop locally, have a staging app with a copy of the production db, and then the production app. I use SVN for version control.
When deploying my production app I have been just moving the urls.py and settings.py files up a directory, deleting my django app directory with rm -rf command and then doing an svn export from the repository which creates a new django app directory with my updated code. I then move my urls.py and settings.py files back into place and everything works great.
My new problem is that I am now storing user uploads in a folder inside of my django app, so I can't just remove the whole app dir anymore or I would loose all of my users files.
What do you think my best approach is now? Would svn export --force work, since it should just be overwriting all of my changed files? Should I take an entirely new approach? I am open to advice?
You may want to watch this presentation by Jacob. It can help you improve your deployment process.
I use Bitbucket as my repo and I can simply perform push on my Dev box and run pull/update on Stage/Prod box. Actually I don't run them manually, I use fabric to do them for me :).
Your could use rsync or something similar to backup your uploaded files and use this backup when you deploy your project.
For deployment you could try to use buildout:
http://www.buildout.org/
http://pypi.python.org/pypi/djangorecipe
http://jacobian.org/writing/django-apps-with-buildout/
For other deployment methods see this question:
Django deployment tools
You can move your files to S3 servers (http://aws.amazon.com/s3/), so you will not ever have to care about moving them with your project.

How do clone a Mercurial repository into a directory that already exists?

I have a client's Django project that I'm developing locally, using Mercurial for version control. I push my local repository to my personal remote server (where I keep all my projects) and then when I come to deploy it (on whichever web server) I clone that respository there from my personal server.
This works fine on most servers (where I have total control) but I have a few projects where I'm deploying on to WebFaction. WebFaction is great, but a little unusual with it's setup, as I need to first declare the Django project as an 'application' through their control panel. This creates a few things automatically, such as an 'apache2', 'myproject', etc folder. It's this same folder though where I want to clone the repository from my personal remote server. Doing the usual hg clone command just doesn't work though as it says the destination folder already exists. There isn't much I can do about the contents of this folder really, so I need to work around this.
I'm not an expert at Mercurial and the only way I could seem to work it out is clone it to another folder and then moving all the contents (including the .hg) into the actual folder I want. This seems silly though...
I'm using Mercurial v1.6.2 (installed through easy_install). Could anyone share some light on this?
Many thanks.
Copying just the .hg dir definitely works, but you could also do a hg init and then hg pull http://remote/repo. A repo that has just been initalized has only the 000000000000000 changeset, so you can pull from any repo without getting the "unrelated repos" warning. This is essentially the same as hg clone --pull with a manual init.
You can copy just the .hg folder, then revert or update to tip. E.g.:
cp -a src/.hg dest/
cd dest
hg up -C
you can either move the folder after the fact, or you can just make a symlink to it. my webfaction directory is actually symlinked, so i know it works fine.
In the main, it looks like you might be trying to use Mercurial as an installation manager which is certainly not its design goal.
If I am reading you correctly, part of your source repository should be something like make deploy which puts the files into their proper places. Put another way, having a repository clone (in .hg) in your deployment directory seems odd and trouble-prone.