I've recently started playing around with Mezzanine, a django-based CMS. I recently just managed to configure Fabric to get it uploading to my host, webfaction.com, as its a bit more involved automatically creating the website on the shared hosting, and I wanted to automate that process.
Altogether, that system uses Fabric for template uploads of config files, and pip + virtualenv for handling the python packages.
However, I just recently read about buildout, and how some people swear by it for deployment, and others don't. See here: Django remote deployment with buildout and Fabric and here: http://labs.creativecommons.org/2011/07/29/not-panicking-switching-to-virtualenv-for-deployment/
While I've googled and found a ton of results for buildout vs. pip, there's not much information about buildout + fabric vs. pip + fabric. It seems like some of the features of buildout (uploading config templates, handling supervisor) can be done through fabric. Can someone tell me the advantages and disadvantages of either approach?
Note: As I'm using shared hosting for the foreseeable future, I can't sudo, which it seems buildout may require for a number of existing recipes.
Summary: Pip only installs python packages and there's more you need to do, obviously. You can do most of the extra work in buildout with the advantage that buildout does it for you both locally and on the server. Fabric has to do less, that way. The drawback is buildout's extra complexity, so if a couple of custom fabric commands is enough for you, that might be preferable for you. So: how does the trade-off work for you?
The long version:
Pip is good at installing python packages for your project. Buildout is good at setting up almost everything for a project (including python packages). That's the difference in goals.
Now... you bring fabric into the mix. With pip+fabric, you can call pip from within fabric to grab all the python packages and then you use fabric itself to set up everything else. An apache/nginx config file, creation of a couple of directories ("var/log/"), etc.
With buildout+fabric, you'll have configured buildout already to do a lot of the things like creating directories and generating files from templates and setting up supervisor and setting up a cronjob to fire up supervisor upon #reboot. So the fabfile has to do less.
So... you swap responsibilities. Everything you can do in buildout, you can do in fabric. Everything you can do in buildout, you can do with custom python (or shell) scripts in combination with pip ("read the README for the extra commands you have to do").
Buildout is a good place to do things if it is an integral part of your project. Think about it like this: if you need it both in production on the server and locally on your development machine, you're better off doing it in buildout. Otherwise you have to run fabric on your local machine, too. You can do it, but...
I use fabric in combination with buildout, myself. Buildout is for setting up the project itself, fabric for everything around it. Some examples:
Actually cloning the buildout from git on the production server.
Git pull (and checkout of the proper tag).
Restarting supervisor.
My suggestion: look on pypi for buildout recipes to see if they are handy for you. Do they save you enough work to make it worthwhile to dive into the extra complexity that a full buildout configuration means? If you don't get enough out of buildout, you might be better off with just fabric+pip and a bunch of custom commands in your fabric file.
Take a look at fabtools which adds a lot of nice buildout functionality into your fabfile. I've worked with all sorts, Chef, Puppet (sledgehammer for a walnut) Ansible and Fabric. I find Ansible great for devops teams who are stuck, but don't want to learn a language, but personally, a well organised Fabric project wins hands down.
Related
I have a django project (a django module/app some other modules that are used from the django one) that uses SQLite. This project is for a University course and now I am asked to supply it in such a way so that it may be installed on some server in our faculty. I'm not the one who's going to install it, and I will not be contacted in case of failure, so I am looking for the easiest, simplest way to supply the project for installation.
I have come across django-jython which supposedly allows one to create WAR files from django projects. However, in the Database Backends section, it says:
SQLite3
Experimental. By now, use it only if you are working on improving it. Or if you are really adventurous.
My overall goal is to deliver this project and I would appreciate any helpful advice. In particular:
Is there another way to pack a django project into a WAR file that supports SQLite?
Is it safe to use SQLite with django-jython in spite of this warning? If so, then how?
Is there any other simple way to pack a django project so that it'll be a piece of cake to install?
If the above answers are "no", then what does it take to change the configuration of the project to use MySQL instead?
You should look into Fabric for easy deployment. I haven't used it myself, but I've heard good things.
I've also had good success quickly and easily setting up servers using Gunicorn with Nginx as a reverse-proxy.
As others have said, using virtualenv, with pip, can quickly
get all your dependencies installed via requirements.txt (from virtualenv).
Some of these blog posts may help:
Tools of the modern Python hacker - virtualenv, pip, fabric
Basic Django Deployment - virtualenv, pip, fabric, rsync
Easy Django Deployment - very quick nginx and gunicorn setup
Edit:
As I reread your post I saw your last bullet point/question. Django is designed to be loosely coupled, meaning that there shouldn't (in most cases) be reasons that one app is dependant on sqlite vs mysql. If you don't need to save the data in the db, changing to MySQL is as easy as starting a mysql server on your machine, and changing the settings.py of your django project. This SO question may help
I'm new to django and my very first project is my blog. I wonder how django developers who use pydev normally synchronize with remote hosting server, updating their sites?
I also would like to know, how do you combine usage of git with a django project? Should I just make a repository for the entire project?
At my company we've got an entire git repository for each project, including the Django sources that are put in the PYTHONPATH for each project, making Django versions project dependant. The folder structure is something like:
/.git
/projectname/app1
/projectname/app2
/projectname/manage.py
/django-lib/django/...
As django-lib is not a Python module, we include both / and /django-lib in the PYTHONPATH. If your project is becoming large, you might want to consider using git submodules on your apps.
We've also setup several servers to support the developers. There's a testing server running a central testing database and a setup including Apache with WSGI to make testing on a real server possible, which sometimes is a bit different then the local manage.py the developers use before committing their changes.
The testing server is updated with the master branch of our git repository. We've made several scripts to allow all developers to do this without letting them login to the server via SSH, but that is just during pre-release. After release, that server will become our staging server, and we'll remove all scripts from it to make it just like our production server.
Every developer has setup their local project to make sure that it communicates with the central testing database, containing several test data. I myself push my changes from the commandline, but you could also use EGit for this.
When we've got a release, we put it in a separate branch, called 'release' (obviously) and the production server will pull only from that branch. This is done via SSH, but I don't really know how your server setup looks like, so I guess that that last step is entirely up to you.
I hope that this has helped you a bit. I won't say that this is the best workflow possible, but it works for us and you should figure out what works for you.
Most experienced Django developers use pip(or distribute) and virtualenv deal with all the python packages you might need for your Django projects (including Django itself).
Personally, all I keep in my projects git repository is a bunch of segregated requirements lists generated by pip :
. ~/Dev/environs/$PROJECT_NAME/bin/activate
pip freeze > ./docs/requirements/main.list
I'm fairly sure most django developers would be familiar with Fabric, which I use for :
streamlining local interaction with git and,
pushing to our central repository,
pulling from our production or test server
touching the wsgi on the relevant server
and pretty much any other kind of task you might find yourself using ssh terminal session for.
For those cases where I need to make changes to someone elses django application in order to make it work or suit our purposes, I :
fork it on github,
clone from my forked repo
make the changes
push it up to my own repo
and provide merge requests to the original repo owner
This way, i have a repo where i can use pip requirement lists to keep pulling from until the original application owner gets their own repo updated.
I'm a strong proponent of version control, and am starting work on a Django project. I've done a few before, and have tried a few different approaches, but I haven't yet found a decent structure that I actually feel comfortable with.
Here's what I want:
a) Source code checked into version control
b) Preferably the environment is not checked into version control (something like buildout or pip requirements.txt is fine for setting up the environment)
c) A reasonable "get a new developer going" story
d) A reasonable deployment story - preferably the entire deployment environment could be generated by a script on the server
It seems to me like someone has to have done this before, but many hours of searching have all led to half-baked solutions that don't really address all of these.
Any thoughts on where I should look?
Look at fabric to manage deployments.
This is what I use to manage servers/deployments with fabric: louis (it is just a collection of fabric commands). I keep a louisconf.py file with each project.
I'd recommend using a distributed VCS (git, hg,...) instead of svn. The reason being that the ease of branching allows for several schemes for deployment. You can have, for example, production and staging branches. Then you enforce that the only merges into production happen from staging by convention.
As for getting developers started quickly you have it right with pip and requirements.txt. I think that also means that you are using virtualenv, but if not that's the third piece. I'd recommend getting a basic README in place. Have the first assignment of each developer that joins a project be to update the README.
The rough way to get someone on board is to have her checkout the code, create a virtualenv, and install the requirements.
I'd recommend having a settings.py file that works with sqlite3 and such that a new developer can use to just get going fast (ie after installing the requirements). However, how you manage the different settings files depends on your project layout. There should be some set of default settings for new developers to use, though.
I keep a projects/ directory in my home directory (on Linux). When I need to start a new project, I make a new, shortly-named (that sufficiently describes the project) dir in projects/; that becomes the root of a new virtualenv (with --no-site-packages) for that project.
Inside that dir (after I've installed the venv, sourced it, and installed the copy of django I'll be working with), I "django-admin.py startproject" a subdir, normally by the same short name. That dir becomes the root of my hg repo (with a quick hg init and ci), no matter how small the project.
If there's any chance of sharing the project with other developers (a project for work, for example), I include a pip requirements.txt at the repo root. Only project requirements go in there; django-debug-toolbar and django-extensions, staples for my dev workflow, are not project requirements, for example. South, when we use it, is.
As for the django project, I normally keep the default settings.py, possibly with a few changes, and add the local_settings convention to the end of it (try: from local_settings import *; except ImportError: pass). My and other devs' specific environment settings (adding django-extensions and django-debug-toolbar to installed apps, for example) go in local_settings.py, which is not checked in to version control. To help a new dev out, you could provide a template of that file as local_settings.py.temp, or some other name that won't be used for any other purpose, but I find that this unnecessarily clutters the repo.
For personal projects, I normally include a README if I plan on releasing it publicly. At work, we maintain Trac environments and good communication to get new devs up to speed on a project.
As for deployment, as rz mentioned, I hear fabric is really good for that kind of automated local/remote scripting, though I haven't really taken the chance myself to look into it.
For the uninitiated, a typical shell session for this might look like the following:
$ cd ~/projects/
$ mkdir newproj
$ cd newproj/
$ virtualenv --no-site-packages .
$ source bin/activate
(newproj)$ pip install django django-debug-toolbar django-extensions
... installing stuff ...
(newproj)$ django-admin.py startproject newproj
(newproj)$ cd newproj/
(newproj)$ hg init .; hg ci -A -m "Initial code"
I have to deploy a Django application onto a SuSE Linux Enterprise 11 system. Corporate rules say I need to deploy using RPMs only. While I can use ./setup.py bdist_rpm for each dependency, it's not really sane, since RPM doesn't record all of the dependencies yet. Therefore I'd have no real advantage in using RPMs and managing dependencies manually is somewhat cumbersome and I would like to avoid it.
Now I had the following idea: While building a package, I could create a virtualenv, install all my dependencies via pip there and then package it up with the rest of the code into one solid RPM.
How sensible is this approach?
I've been using this approach for about a year now and it has worked out pretty well.
One gotcha is that you'll want to check out the bang lines in any python scripts written to the virtualenv's bin directory. These will end up being full path names used in your build environment, which probably won't be the same directory where you end up installing the virtualenv. So you may need to add some sed calls in your RPM's postinstall to adjust the paths.
Is there a way to run Pinax without virtualenv?
I want to run it without virtualenv as I want to run it on a django-container on mediatemples grid-hosting service. Their containers can scale upto 1Gb of dedicated memory, so I wouldnt have to worry about my own VPS or scaling issues. But their response was:
" because of the way the DjangoContainer works, you won't be able to configure your server to use your virtualenv. Essentially the DjangoContainer is a virtualized server (to which you don't have access other than the AccountCenter tools, or the 'mtd' command line tool) with the specific purpose of serving your Django applications. It mounts your django container folder so that it has your application code, but you cannot modify the version or location of python it uses. This probably means you'll have to use Pinax without virtualenv support, as the general idea of using virtualenv in this way would be to create a custom environment for your Pinax application, which as I mentioned here is impossible to instruct the server to use. "
As of 0.9a1, Pinax can be used without pinax-boot.py which was the virtualenv dependency (we bundled it). Requirements are project-level and must be installed with pip. However, setup_project does enforce a virtual environment when installing requirements (it calls pip for you as a convenience; I would be open to not enforcing a virtual environment here). You can pass --no-reqs to setup_project forcing it to skip dependency installation. You can then run pip yourself and install it however you like.
technically yes, but you would have to change out quite a bit of the configuration that is handed out and hand install a lot of libraries. Pinax has virtualenv as a very low level built in assumption.
you can, all you need to do is find out what is in the virtualenv. set it up and install yolk in the virtual env and type yolk -l to see what you need to install to get it to work.