I have built a software running on an ubuntu 14.04, it runs on a python environment with a lot of dependencies (databases, flask, etc...).
I am trying to figure out the best way to have an easily and portable environment for migration. For example, if I want to install the software on another computer or maybe AWS...
I checked out some solutions and for now, Vagrant, Puppet, Chef look interesting but I am a bit confused.
Furthermore, the software requires high performances and it would be way too slow to run it on a VM.
What are your thoughts on that ? Basically an easy way to install a software with all his dependencies and being able to migrate the environment on whatever host I would need.
Thanks a lot
Related
Hopefully this isn't too stupid of a question, concerning the use of pipenv for the same Django project on different computer systems. The scenario is that I'm using pipenv with a test Django project on one laptop, everything works fine, using VS Code and it's using the proper pipenv environment for the Python environment within VS Code.
The project however is within Dropbox so when I'm using a different laptop, which I do sometimes, one is my work laptop the other is my personal one at home, I can work on the same project wherever I left off. So you can probably deduce the issue I'm having.
I'm using pipenv environment A on my work laptop for the Django project. But when I open the project in VS Code on my personal laptop at home I have to keep specifying the proper pipenv environment to use, which obviously is different than the one on my work laptop.
Maybe I shouldn't be working this way and should just work on one laptop for the project, but I imagine others have done the same with similar setups before.
Is there a "proper" way to do this, using different pipenv environments on different machines for the same Django project? Should I copy over the pipenv environment to each machine as well? Or am I missing something really simple here?
First, use version control. (git is the defacto standard these days and there are plenty of services to choose from)
For using pipenv, you want to keep your Pipfile and Pipfile.lock in your repository (these are the only files you should be copying related to your environment from system to system).
The local environment is then built on each machine where it is needed by using pipenv sync or pipenv sync --dev (to install dev specific requirements).
So the related virtualenv will be created on each system in which you are developing your project.
I'm developing a Django web app with Channels. While I'm following this tutorial , it is required to install Docker.
I'm working on WSL on windows 10 HOME, and so, it is really painful to install Docker.
I just discover Docker, I'm a little confuse about it, I understand it is a tool which facilitates the deployment of a web app on a web hosting later. But I'm not sure.
Could you give me your advice ? Could you tell me if it is really important to use Docker for my project ?
Would Have I less pain if I would develop on a Ubuntu OS ?
Thank you,
The following are my own considerations, not pretending to be exhaustive Docker review.
Moving to Docker would give you following advantages:
Easy deploy - you don't need to supply manuals on how to install your app, dependencies and link them together. Only How to install Docker (btw for Windows it hurts:)
Isolation - your services get isolated network and do not bother the host
Easy upgrade - just push new image and that's it
Decomposition - with docker-compose and other tools you will be able to split your application into services and maintain them separately
Scaling - with proper design, tools like k8s will allow you to easily scale app by adding replicas of your services
From the other hand, on Windows Docker create additional overhead, unlike Linux where it is implemented on top of Linux kernel, also you need Win10 Professional to enjoy Docker and not docker toolbox.
Also Windows is not so good in automated package management and installing software for Windows in many cases cannot be done as simple as apt-get install whatever, thus you loose another Docker benefit - easy system preparation via Dockerfile.
If you plan to stay only on Windows, based on my own exp I would probably not recommend moving to Docker, because I personally found it difficult to use without VirtualBox/Ubuntu.
I've recently started playing around with Mezzanine, a django-based CMS. I recently just managed to configure Fabric to get it uploading to my host, webfaction.com, as its a bit more involved automatically creating the website on the shared hosting, and I wanted to automate that process.
Altogether, that system uses Fabric for template uploads of config files, and pip + virtualenv for handling the python packages.
However, I just recently read about buildout, and how some people swear by it for deployment, and others don't. See here: Django remote deployment with buildout and Fabric and here: http://labs.creativecommons.org/2011/07/29/not-panicking-switching-to-virtualenv-for-deployment/
While I've googled and found a ton of results for buildout vs. pip, there's not much information about buildout + fabric vs. pip + fabric. It seems like some of the features of buildout (uploading config templates, handling supervisor) can be done through fabric. Can someone tell me the advantages and disadvantages of either approach?
Note: As I'm using shared hosting for the foreseeable future, I can't sudo, which it seems buildout may require for a number of existing recipes.
Summary: Pip only installs python packages and there's more you need to do, obviously. You can do most of the extra work in buildout with the advantage that buildout does it for you both locally and on the server. Fabric has to do less, that way. The drawback is buildout's extra complexity, so if a couple of custom fabric commands is enough for you, that might be preferable for you. So: how does the trade-off work for you?
The long version:
Pip is good at installing python packages for your project. Buildout is good at setting up almost everything for a project (including python packages). That's the difference in goals.
Now... you bring fabric into the mix. With pip+fabric, you can call pip from within fabric to grab all the python packages and then you use fabric itself to set up everything else. An apache/nginx config file, creation of a couple of directories ("var/log/"), etc.
With buildout+fabric, you'll have configured buildout already to do a lot of the things like creating directories and generating files from templates and setting up supervisor and setting up a cronjob to fire up supervisor upon #reboot. So the fabfile has to do less.
So... you swap responsibilities. Everything you can do in buildout, you can do in fabric. Everything you can do in buildout, you can do with custom python (or shell) scripts in combination with pip ("read the README for the extra commands you have to do").
Buildout is a good place to do things if it is an integral part of your project. Think about it like this: if you need it both in production on the server and locally on your development machine, you're better off doing it in buildout. Otherwise you have to run fabric on your local machine, too. You can do it, but...
I use fabric in combination with buildout, myself. Buildout is for setting up the project itself, fabric for everything around it. Some examples:
Actually cloning the buildout from git on the production server.
Git pull (and checkout of the proper tag).
Restarting supervisor.
My suggestion: look on pypi for buildout recipes to see if they are handy for you. Do they save you enough work to make it worthwhile to dive into the extra complexity that a full buildout configuration means? If you don't get enough out of buildout, you might be better off with just fabric+pip and a bunch of custom commands in your fabric file.
Take a look at fabtools which adds a lot of nice buildout functionality into your fabfile. I've worked with all sorts, Chef, Puppet (sledgehammer for a walnut) Ansible and Fabric. I find Ansible great for devops teams who are stuck, but don't want to learn a language, but personally, a well organised Fabric project wins hands down.
This is a Django and Python and maybe just a general web development question.
what's the difference between using virtualenv vs vagrant vs virtual box and etc... ?
I'm kinda confused as to when to use which one :/ I've been using virtual env this whole time and creating new virtual environments for different projects....
Is this the right way to do it?
One virtualenv per project?
I'm not really sure when and where vagrant comes into play...Am I supposed to set up vagrant and then use virtualenv?
This is probably a silly question but...if I were to be working on this project with other people. Would they too have to set up a virtual env? Just to collaborate?
Wouldn't it make more sense that we all work on our local machines and then push it into the main branch? I'm just kinda confused .... I feel like I'm doing it all wrong...
Thanks for the replies everybody!
Virtualenv sets up a local sandbox for you to install Python modules into.
Vagrant is an automation tool for creating Virtual Machines.
VirtualBox is a free, open source environment for running virtual machines, like those created by Vagrant.
Virtualenv is really about all you'll need to do sandboxed development on your local machine. We use Vagrant at my work to automate the creation of VMs. This way new developers coming on to a project have basically zero configuration to do in order to start working.
If you're collaborating with other devs, they don't need to do any of the above to work on your Django project, but if there's a lot of configuration involved that can't be done with pip and a requirements.txt, then you might look at Vagrant to ease some of that automation.
But you are correct in your assumption that you can all just work on a local branch and push back to the repo. Everything else is just icing.
Virtualenv is a python construct that holds a specific set of packages, separate from your system packages. The version of Python and its packages that came with your OS or that you installed separately is a "system package".
Virtualbox is totally different -- it's a VM, an entire operating system in a box.
I'm not familiar with Vagrant.
All you need is virtualenv. Create a new virtualenv for each project (they're very lightweight!) You need to do this because the whole point of virtualenv is to isolate the exact packages and versions of those packages you need for your project. Then activate the virtualenv and use pip install to install the packages you need, presumably starting with Django itself.
Once you have all the packages you need, use pip freeze > requirements.txt to create a file called requirements.txt that records all of the packages you've decided to use.
When other people collaborate on your project, they can start a virtualenv, pull your code into it, and run pip install -r requirements.txt to replicate your environment. They can even modify requirements.txt, push that back to you via your version control system, and you can run pip install -r requirements.txt yourself to modify your environment to match their changes.
This is all essential because without virtualenv, the problem of, for instance, having one project on your computer that requires Django 1.4 and one that requires Django 1.5 becomes very complicated.
Virtualenv is not an entire operating system in a box, just a python environment, so even if you are using it, you are still working on your local machine.
We use virtualenv and a Ubuntu virtual machine. Here's why:
virtualenv allows us to have isolated Python environments on a given operating system instance
Using Ubuntu dekstop in a virtual machine for our Python development mimics what it will look like when deployed on the server which is also Ubuntu. This means that we understand precisely the external OS package dependencies and configuration. You don't get this easily when you use OSX or Windows for development and Linux for deployment.
One important point is that a virtual machine is portable. You can take a snapshot and deploy it elsewhere easily. With Vagrant and Ansible combination you can automate a remote deployment.
I'm installing Solr to be used with Haystack / Django.
How do I go about installing Solr? Do I need to follow the online instructions exactly and install services like Tomcat?
The above questions I just commented make all the difference if you are set on installing / setting this up yourself. OR you could use a hosted service (read: easy).
Try out http://websolr.com I'm loving it.
If you are on OSX and want a local Solr instance just to try out for some development, this is an awesome tutorial: http://realityloop.com/blog/2011/07/19/setting-multicore-apache-solr-os-x-using-homebrew
essentially brew install solr, and then some configuration.
If you are trying to do a production level solr server on a linux operating system, I would suggest working with an instruction set specific to that distribution and the version you are using. OR even better, bite the bullet and pay someomne who knows what they are doing to set it up (make sure your indexes won't fail or go down).