I'm looking at setting up a continuous integration solution for a Django framework project hosted in AWS. Their are client and server tests. The client tests will fire up a headless browser, but it would also be good if it could fire up real browsers like Chrome and Firefox.
Repo hosted on BitBucket. What would be great is if code is committed to a branch, then the CI setup will pull the changes and run the tests in an environment that closely matches production.
Can you help me set up the best possible stack given the above constraints?
I think Fabric would be suitable for this. Fabric allows you to run code on your local machine and the server. Once you learn the basics of Fabric, you will see that you can run a command like 'fab deploy' and Fabric will commit your changes to a branch, then pull the changes onto the server.
You could have steps in between to launch a new server, prepare the environment, then run tests against the code. I have been using Fabric to simplify some deployments and have found it a great experience. Because of the flexibility fabric gives you, you can pretty much deploy however you want, whether it's onto a test server, QA, live etc...
Related
I want to work locally on my django(1.7) project and regularly deploy updates to a production server. How would you do this? I have not found anything about this in the docs. I am confused about that because it seems like many people would want to do that and there should be some kind of standard solution to this. Or am I getting the whole workflow wrong?
I should note that I'm not expecting a step-by-step guide. I am just trying to understand the concept.
Assuming you already have your deployment server setup, and all you need to do is push code to your server, then you can just use git as a form of deployment.
Digital Ocean has a good tutorial at this link https://www.digitalocean.com/community/tutorials/how-to-set-up-automatic-deployment-with-git-with-a-vps
Push sources to a git repository from a dev machine.
pull sources on a production server. Restart uwsgi/whatever.
There is no standard way of doing this, so no, it cannot be included with Django or be thoroughly described in the docs.
If you're using a PaaS how you deploy depends on the PaaS. Ditto for a container like docker, you must follow the rules of that particular container.
If you're old-school and can ssh into a server you can rsync a snapshot of the code to the correct place after everything else is taken care of: database, ports, webserver setup etc. That's what I do, and I control stuff with bash scripts utilizing a makefile.
REMOETHOST=user#yourbox
REMOTEPATH=yourpath
REMOTE=$REMOTEHOST:$REMOTEPATH
make rsync REMOTE_URI=$REMOTE
ssh $REMOTEHOST make -C $REMOTEPATH deploy
My "deploy"-action is a monster but might be as easy as something that touches the wsgi-file used in order to reload the site. My medium complex ones cleans out stale files, run collectstatic and then reloads the site. The really complex ones creates a timestamped virtualenv, cloned database and remote code tree, a new server-setup that points to this, runs connection tests on the remote and if they succeed, switches the main site to point to the new versioned site, then emails me the version that is now in production, with the git hash and timestamp.
Lots of good solutions. Heroku has a good tutorial: https://devcenter.heroku.com/articles/getting-started-with-django
Check out a general guide for deploying to multiple PaaS providers here: http://www.paascheatsheet.com
I am currently evaluating the possibility of introducing a private Java PAAS cloud. So far I am quite excited about the whole solution, especially combining the foundry with openstack.
What I am wondering though, is how this can be combined with development. I obviously want the developer to run the developed code on the cloud and no longer on his unmanaged workstation.
Is it possible to do the following:
Developer develops his application code on the local host OS. A virtual machine is used to build and run the application. I have seen this in vagrant and liked this alot. Ideally the local vagrant box is a cloud foundry space.
If the developer is OK with his code, he should push his application out of the local vm to a developer specific acceptance space run by cloud foundry on the network. Here the application is a more production like environment and automated acceptance / disaster recovery tests can be executed.
If the developer decides this is OK and merges his changes to the trunk (SVN/GIT) a CI tool should deploy the application to the "global" test, acceptance and production spaces.
I assume the last point is no problem. I just cannot find a way, how the first steps can be achieved.
Any ideas?
are you actually looking for a complete cf deployment on top of openstack?
That can be achieved using BOSH cloud foundry deployment for openstack.
http://docs.cloudfoundry.com/docs/running/deploying-cf/openstack/
you can have different spaces in the cf deployment: test , production etc. and can move application from one space to another after testing is done.
Recently, I have started to deploy my work-in-progress django site from my local to server. But I have been doing it manually, which is ugly, unorganized, and error-prone.
I am looking for a way to automate and streamline the following deployment tasks:
Make sure all changes are committed and pushed to remote source repository (mercurial) and tag the release.
Deploy the release to the server (including any required 3rd-party apps missing from the server)
Apply the model changes to the database on the server
For 2), I have two further questions. Should the source of the deployment be my local env or the source repository? Do I need a differential or full deployment?
For 3), I use South in my local for applying model changes to database. Do I do the same on the server? If so, how do I apply multiple migrations at once?
I think Fabric is the defacto lightweight python deployment tool. http://docs.fabfile.org/en/1.3.4/index.html. It is very simple and will help you keep your deployment organized and streamlined. It allows for easy scp or rsync. Additionally it is easy to integrate with django tests.
For my smaller projects I just make the source of my deployments my local env. I checkout a clean copy and deploy from there. It would probably be better to integrate this with my version control for a quick rollback if there are any errors once I deploy.
I have never used south, but i'd imagine you could just write a fab command to sync your production server. If you're using south on dev, i couldn't imagine why you wouldn't want to use it on production too?
I have never actually worked for a company which is deploying a Django App (with a large user base), and am curious about what is the best way to do this.
Right now I am hosting a Django App on EC2. The code for the app is sitting in my github account. I have nginx serving static content, and behind it a single apache server running django + mod_wsgi.
I am trying to figure out what the best practice is for "continuous deployment". Right now, after I have added additional functionality I do the following on EC2:
1) git reset HEAD --hard
2) git pull
3) restart apache
4) restart nginx
I have custom logic in my settings.py file so that if I am running on EC2, debug gets set to False, and my databases switch from sqlite3 (development) to mysql (production).
This seems to be working for me now, but I am wondering what is wrong with this process and how could I improve it.
Thanks
I've worked with systems that use Fabric to deploy to multiple servers
I'm the former lead developer at The Texas Tribune, which is 100% Django. We deployed to EC2 using RightScale. I didn't personally write the deployment scripts, but it allowed us to get new instances into the rotation very, very quickly and scales on-demand. it's not cheap, but was worth every penny in my opinion.
I'd agree with John and say that Fabric is the tool to do this sort of thing comfortably. You probably don't want to configure git to automatically deploy with a post commit hook, but you might want to configure a fabric command to run your test suite locally, and then push to production if it passes.
Many people run separate dev and production settings files, rather than having custom logic in there to detect if it's in a production environment. You can inherit from a unified file, and then override the bits that are different between dev and production. Then you start the server using the production file, rather than relying on a single unified settings.py.
If you're just using apache to host the application, you might benefit from a lighter weight solution. Using fastcgi with nginx would allow you to do away with the overhead of apache entirely. There's also a wsgi module for nginx, but I don't know if it's production ready at this point.
There is one more good way how to manage this. For ubuntu/debian amis it is good to manager versions and do deployemnts by packeging your application into .deb
I am researching about how to set up CI and continuous deployment for a small team project for a Django based web application. Here are needs:
Developer check in the code into a hosted SVN server (unfuddle.com)
A CI server detects new checkin, check out the source, build, run functional tests.
If tests all passed, deploy the code to the webserver on Amazon EC2.
For now, the CI server is also responsible to run the functional tests. I figured out that I can use Husdon as the CI server, use Selenium to run functional tests, and use Fabric to deploy the build to remote web server in Amazon cloud.
I am new to Django development and not very familiar with opensource tools. My questions are:
I can find some information to integrate hudson with selenium, but I couldn't find much information on how to integrate Fabric to Hudson as well. Is this setup viable? Do you see problems?
How do I integrate and deploy database changes? Most likely in the early stage we will change database schema very often with code changes. I used to use Visual Studio and the database project made it very simple to deploy. I wonder if there is "established, well-supported" way to do that.
Thanks!!
Can't help you very much with continuous integration/deployment - at my work we used CruiseControl and it acted very much as you describe, but I didn't have anything to do with setting it up.
However in answer to your second question, on database changes, the current state of the art in Django is to use South, which has just released version 0.7. It works well in a CI environment, as db migrations are stored alongside the code for each app, so the CI server can be set to run them automatically before running the tests.