We have an app that works perfectly fine on production but very slow on the dev machine.
Django==2.2.4
I'm using Ubuntu 20.04 but other devs are using macOS and even Windows.
Our production server is a very small compared to the dev laptops (it works very slow on every dev environment, we are 5 developers).
The app makes several request since it's a Single Page application that uses Django Rest Framework and React.js in the front-end.
We have tried different databases locally (currently postgresql, tried MySQL and sqlite3), using docker, no docker, but it does not change the performance.
Each individual request takes a few seconds to execute, but when they go all toghether the thing gets very slow. As more request are executed, the performance starts to drop.
It takes the app between 2/3 minutes to load in the dev environment and in any production or staging environment it takes little above 10 seconds.
Also tried disabling DEBUG in the back and front-end, nothing changes.
It is my opinion that one of the causes is that the dev server is single thread and it does not process a request until the previous is finished.
This makes the dev environemnt very hard to work with.
I've seen alternatives (plugins) to make the dev server multi-thread but those solutions do not work with the latests versions of django.
What alternatives could we try to improve this?
Looks like posting this question helped me think in an alternative. Using gunicorn in the dev environment really helps.
Installed it with
pip install gunicorn
And then execute it using this:
venv/bin/gunicorn be-app.wsgi --access-logfile - --workers 2 --bind localhost:8000
Of course if I want to access the static and media files I'll have to set up a local nginx but it's not a big deal
Related
I generally deploy django web applications in Ubuntu.
But currently we are using cpanel(not looking for alternatives) which doesnt work in Ubuntu. so want to know if moving to centos for cpanel is worth ?
Because my fear is, if we move to centos server, do we have to face some complex issues(with django deployment) that we might take lot of time(we are good at Ubuntu).
Deploying on centos is essentialy the same. You will probably want to run django in gunicorn and supervisor watching over the process. And nginx for serving static files.
I have done deployment on CentOS aswell as on Ubuntu and I didn't encounter any problems.
You can use the below link to help you in deployment
https://www.digitalocean.com/community/tutorials/how-to-redirect-www-to-non-www-with-nginx-on-centos-7
and if you had any issues in permission you can disable SE-LINUX
I have a django application running on a server. I want to use let's encrypt to provide an encrypted connection. I could use the standalone option of their ACME client, but i don't want to stop my server, what i would have to do.
So there is the webroot option, that work with my allready running webserver (nginx). Django would process the request in this case. My question is, how should it look like on the django side to get this running (keeping automated renewal several months in mind)?
I don't know what setup others use, but I generally set up Django apps with Nginx serving static content and Gunicorn as the application server. It's widely accepted that Django apps usually use this kind of two web server setup. The standard instructions for setting up Let's Encrypt with Nginx worked fine for me.
Or Digital Ocean have an excellent guide too.
EDIT: It looks like Nginx can do a "graceful" reload that just updates the config with no downtime. For Debian or Ubuntu pre Systemd this would be sudo service nginx reload, while for a distro with Systemd the command is sudo systemctl reload nginx.service.
In case other users come this way like I did from Google, here's how I improved this situation:
I was unsatisfied by my options when it came to creating ACME challenges for Let's Encrypt when running a Django application. So, I rolled my own solution and created a Django app! Basically, you can manage your ACME challenges as just another object, and the app will produce the proper end-point URL.
Yes you are installing an app which means a deploy / update to your app, but once you've done that managing your challenges is far easier in the long run.
Simply pip install django-letsencrypt and follow the README to be on your way.
Currently I have 3 linode servers:
1: Cache server (Ubuntu, varnish)
2: App server (Ubuntu, nginx, rabbitmq-server, python, php5-fpm, memcached)
3: DB server (Ubuntu, postgresql + pg_bouncer)
On my app-server I have multiple sites (topdomains). Each site is inside a virtualenviroment created with virtualenvwrapper. Some sites are big with a lot of traffic, and some site are small with little traffic.
A typical site consist of python (django), celery (beat, flower) and gunicorn.
My current development pattern now is working inside a staging environment on the app-server and committing changes to git. Then change environment to the production environment and doing a git pull, and a ./manage.py migrate and restarting the process doing sudo supervisorctl restart sitename:, but this takes time! There must be a simpler method!
Therefore it seems like docker could help simplify everything, but I can't decide the best approach for how I could manage all my sites and containers inside each site.
I have looked at http://panamax.io and https://github.com/progrium/dokku, but not sure if one of them fit my needs.
Ideally I would run a development version of each site on my local machine (emulating cache-server, app-server and db-server), do code changes there and test them. When I would see the changes worked, I would execute a command that would do all the heavy lifting and send the changes to the linode servers (I would think mostly the app-server), do all the migration and restarting the project on the server.
Could anyone point me in the right direction as how to achieve this?
I have faced the same problem. I don't claim this is the best possible answer and am interested to see what others have come up with.
There doesn't seem to be any really turnkey solution on Docker yet.
It's also been frustrating that most of the 'Django+Docker' tutorials just focus on a single Django site, so they bundle up the webserver and everything in the same Docker container. I think if you have multiple sites on a server you want them to share a single webserver, but this quickly gets more complicated than presented in the tutorials, which are no longer much help.
Roughly what I came up with is this:
using Fig to manage containers and complicated Docker config that would be tedious to type as commandline options all the time
sites are Django, on uWSGI+Nginx (no reason you couldn't have others though)
I have a git repo per site, plus a git repo for the 'server'
separate containers for db, nginx and each site
each site container has it's own uWSGI instance... I do some config switching so I can either bring up a 'dev' container with uWSGI as acting standalone web server, or a 'live' container where the uWSGI socket is exposed to the main Nginx container, which then takes over as front-side web server.
I'm not sure yet how useful the 'dev' uWSGI servers are, I might switch to just running Django dev server and sharing my local code dir as a volume in the container, so I can edit and get live reloading.
In the 'server' repo I have all the shared Dockerfiles, for Nginx server, base uWSGI app etc.
In the 'server' repo I have made Fabric tasks to do my deployment (checkout server and site repos on the server, build docker images, run fig up etc).
Speaking of deployment, frankly I'm not much keen on the Docker Registry idea. This seems to mean you have to upload hundreds of megabytes of image file to the registry server each time you want to deploy a new container version. This sucks if you are on a limited bandwidth connection at the time and seems very inefficient.
That's why so far I decided to deploy new code via Git and build the new images on the server. I don't use a Docker Registry at all (apart from the public one for a base Ubuntu image). This seems to go against the grain of Docker practice a bit so I'm curious for feedback.
I'd strongly recommend getting stuck in and building your own solution first. If you have to spend time learning a solution like Dokku, Panamax etc that may or may not work for you (I don't think any of them are really ready yet) you may as well spend that time learning Docker directly... it will then be easier to evaluate solutions further down the line.
I tried to get on with Dokku early on in my search but had to abandon because it's not compatible with boot2docker... which means on OS X you're faced with the 'fun' of setting up your own VirtualBox vm to run the Docker daemon. It didn't seem worth the hassle of this when I wasn't certain I wanted to be stuck with how Dokku works at the end of the day.
I want to deploy a django site (it is the open source edx code on github).
I am faced with choosing between using
Apache with mod_wsgi
nginx with gunicorn
I have used Apache with mod_wsgi and it's cool enough, but i have no experience with the second option.
Which of these would be a better option in terms of speed and also to some extent, ease of use?
NB: I would need to run two different django sites on say, port 80 and 81 and access them from two different subdomains.
Nginx is a really light and easy to use solution and along with gunicorn it allows us to run any wsgi application and scale it easily.
Nginx is better at handling requests since it does not spawn a new process for every request unlike Apache.
I have written an answer on how to deploy django with nginx for a related question:
Deploying Django project with Gunicorn and nginx
Well,the few milliseconds you get with Nginx will not make a hudge difference regarding the time other processes take. Nginx may save RAM but it would only be a great difference on servers with a few RAM. For specific uses on big website there could be some more notable differences but this will become an expert affair then.
The real difference for most is probably the ease of learning. I don't find Apache to be specifically hard to use and the doc is clean. However most of Python tutorials I found are about using Nginx with Gunicorn.
If you already know how to use Apache with Python it would probably be more straight to the point to use it, unless you want to learn Nginx too to improve your CV.
However, if you are a newcomer, there is more documentation about Nginx with Python. It makes it the easier option.
I have good experience with nginx and gunicorn. They keep on working great when I've finally set all the settings right and got it running.
For nginx and gunicorn they are:
* nginx configuration files (/etc/nginx/sites-enabled/ and /etc/nginx/nginx.conf)
* gunicorn configuration files (/etc/init/gunicorn.conf and /etc/gunicorn.d/)
I've seen a tutorial for apache + mod_wsgi and it seems so much simpler to set up.
I have primarily worked with nginx and gunicorn. I am currently working with apache + mod_wsgi. It is actually easy if your Python version is 2.7 because mod_wsgi when installed directly from the package manager will work normally. But if your code is in a different Python version. mod_wsgi has to be built from source with the same version. If you installed your Python also from source then the procedure to get the whole web application working is fairly difficult.
Nginx and gunicorn on the other hand do not have any version issues, since the proxypass param makes it easy to forward requests to gunicorn. All we need to ensure is that gunicorn is running with the same version of Python that your code is in.
I am quite a noob when it comes to deploying a Django project. I'd like to know what are the various methods to deploy Django project and which one is the most preferred.
The Django documentation lists Apache/mod_wsgi, Apache/mod_python and FastCGI etc.
mod_python is deprecated now, one should use mod_wsgi instead.
Django with mod_wsgi is easy to setup, but:
you can only use one python version at a time [edit: you even can only use the python version mod_wsgi was compiled for]
[edit: seems if I'm wrong on mod_wsgi not supporting virtualenv: it does]
So for multiple sites (targeting different django/python versions) on a server mod_wsgi is not
the best solution.
FastCGI can be used with virtualenv, also with different python versions, as you run it with
./manage.py runfcgi …
and then configure your webserver to use this fcgi interface.
The new, hot stuff about django deployment seems to be gunicorn. It's a webserver that implements wsgi and is typically used as backend with a "big" webserver as proxy.
Deployment with gunicorn feels a lot like fcgi: you run a process doing the django processing stuff with manage.py, and a webserver as frontend to the world.
But gunicorn deployment has some advantages over fcgi:
speed - I didn't find the sources, but benchmarks say fcgi is not as fast as the f suggests
config files, for fcgi you must do all configuration on the commandline when executing the manage.py command. This comes unhandy when running multiple django instances via an init.d (unix-like OS' system service startup). It's always the same cmdline, with just different configuration files
gunicorn can drop privileges: no need to do this in your init.d script, and it's easy to switch to one user per django instance
gunicorn behaves more like a daemon: writing pidfile and logfile, forking to the background etc. makes again using it in an init.d script easier.
Thus, I would suggest to use the gunicorn solution, unless you have a single site on a single server with low traffic, than you could use the wsgi solution. But I think in the long run you're more happy with gunicorn.
If you have a django only webserver, I would suggest to use nginx as frontendproxy, as it's the best performing (again this is based on benchmarks I read in some blogposts - don't have the url anymore).
Personally I use apache as frontendproxy, as I need it for other sites hosted on the server.
A simple setup instruction for django deployment could be found here:
http://ericholscher.com/blog/2010/aug/16/lessons-learned-dash-easy-django-deployment/
My init.d script for gunicorn is located at github:
https://gist.github.com/753053
Unfortunately I did not yet blog about it, but an experienced sysadmin should be able to do the required setup.
Use the Nginx/Apache/mod-wsgi and you can't go wrong.
If you prefer a simple alternative, just use Apache.
There is a very good deployment document: http://lethain.com/entry/2009/feb/13/the-django-and-ubuntu-intrepid-almanac/
I myself have faced a lot of problems in deploying Django Projects and automating the deployment process. Apache and mod_wsgi were like curse for Django Deployment. There are several tools like Nginx, Gunicorn, SupervisorD and Fabric which are trending for Django deployment. At first I used/configured them individually without Deployment automation which took a lot of time(I had to maintain testing as well as production servers for my client and had to update them as soon as a new feature was tested and approved.) but then I stumbled upon django-fagungis, which totally automates my Django Deployment from cloning my project from bitbucket to deploying on my remote server (it uses Nginx, Gunicorn, SupervisorD, Fabtic and virtualenv and also installs all the dependencies on the fly), all with just three commands :) You can find more about it in my blog post here. Now I even don't have to get involved in this process(which used to take a lot of my time) and one of my junior developers runs those three commands of django-fagungis mentioned here on his local machine and we get a crisp new copy of our project deployed in minutes without any hassle:)