Django Download/Upload Files In Production - django

Question

I have a Django project that is currently hosted. I am serving static files but do not know how to handle user file uploads / downloads in the MEDIA folder. I've read a lot about Docker, Nginx, and Gunicorn but have no idea which I need nor how to set them up. I've followed no less than 20 tutorials and watched no less than 15 YouTube videos but am still confused (I've visited every link for the first 2 pages of all my Google searches).
My question is, which of these do I need to allow users to upload/download files from a site? On top of that, I have tried getting all three working but can't figure them out, surely I'm not the only one that has so much difficulty with this, is there a good resource/tutorial which would guide me through the process (I've spent well over 40 hours reading about this stuff and trying to get it to work, I've gotten to the point where so long as it works, I don't care for understanding how it all fits together)?
Thank you.
edit - this is a stripped down version of what was requested. I haven't included the html as it's my first time doing this and I've used Ajax and things and it's a complete mess and I'm sure will just confuse you.
settings.py
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static_files')
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'static_media')
STAT=os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = [ STAT,
os.path.join('static/'),
os.path.join('templates/static/'), # for overridden templates css
]
view.py
class UploadView(View):
def get(self, request):
files_list = UploadedFile.objects.all()
return render(self.request, 'users/modules.html', {'files': files_list})
def post(self, request, *args, **kwargs):
data = {}
form = UploadedFileForm(self.request.POST, self.request.FILES)
form.instance.data_id = self.request.POST['data_id']
if form.is_valid():
uploaded_file = form.save()
Thank you.

Answer 1

So you want to know how to make your (working in development environment) project production-ready. Let's start with what components are required
Web Server (Nginx)
Application Server (uWSGI)
Application (Django)
Web Server serves the users' requests. It knows how to generate the correct output for a request - read a file from a filesystem, pass a request further to application server and so on. Nginx is the good choice.
Application Server is the middleman between Web Server and Application. It can spawn application instances (processes), balance the load between those instances, restart dead instances and many other things. uWSGI is good choice here.
Application - in your case it's Django project you have working in your development environment. You have everything ready here, but most likely you should adjust settings a bit. Django will communicate with Application Server through WSGI protocol.
At this point you should also understand how a web browser will load, render and display your site. All starts from a user who wants to open a page on your site, for example http://example.com/uploads. A browser will send HTTP GET request to your server and Web Server program (Nginx) will catch this request and decide what to do next.
Since that particular request isn't about some static file (static HTML file or static JPEG image and so on) - Nginx will decide to pass a request to Application Server. uWSGI will get the request and pass it forward to Django.
Django will use all the urls.py files to find the right view to generate the response for http://example.com/upload page. What your view will do?
def get(self, request):
files_list = UploadedFile.objects.all()
return render(self.request, 'users/modules.html', {'files': files_list})
it will return the HTML page (rendered template). So that HTML document will be returned back to Application Server, then back to Web Server and finally to a user's web browser.
Then a browser will start parsing of that HTML document and most likely it will find some additional resources to load - css, javascript, images, fonts, ... For each resource - it will make additional GET request to Web Server. And this time Web Server will not push requests forward to Application Server. It will just read those files from file system and return back to a browser.
Those resources are not dynamic, they are static. So you basically store them under static namespace. For example:
http://example.com/static/main.css
http://example.com/static/main.js
http://example.com/static/logo.png
...
Those files are the part of your application. Your application is shipped with those files. But also there're some files which could be uploaded to your system. You could save those files in any place - filesystem, database, ... But you must have URLs for them. Usually it's media namespace:
http://example.com/media/user-file.csv
http://example.com/media/reports/john-02-12-2020.pdf
...
In both cases - those calls will be just forwarded to Web Server and then to filesystem.
Why do you see everything working on your development environment? It's likely because you run the application with python manage.py runserver. In that case - Django is your Web Server as well (and there will be no Application Server middleman). So it will manage it's own instances, it will get user's requests, it will return static files, it will return "media" files, it will return dynamically generated pages and so on.
Each component described above needs it's own configuration file. Let me show you some examples you can use for your project.
Web Server (Nginx)
sites-enabled/default.conf
upstream uwsgi {
server uwsgi:8080;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
error_log /var/log/nginx/error.log;
charset utf-8;
location /media {
alias /home/web/media;
expires 7d;
}
location /static {
alias /home/web/static;
expires 7d;
}
location / {
uwsgi_pass uwsgi;
include uwsgi_params;
}
}
Notes:
everything under http://example.com/static will go to filesystem (/home/web/static directory)
for example: http://example.com/static/css/main.css -> /home/web/static/css/main.css
everything under http://example.com/media will go to filesystem (/home/web/media directory)
for example: http://example.com/media/reports/john-02-12-2020.pdf -> /home/web/media/reports/john-02-12-2020.pdf
everything else will be passed to uWSGI (to host uwsgi, to port 8080)
Application Server (uWSGI)
uwsgi.conf
[uwsgi]
req-logger = file:/var/log/uwsgi-requests.log
logger = file:/var/log/uwsgi-errors.log
workers = %k
# enable-threads = true
# threads = 4
chdir = /home/web/app
module = core.wsgi
master = true
pidfile=/tmp/app.pid
socket = 0.0.0.0:8080
env = DJANGO_SETTINGS_MODULE=core.settings
memory-report = true
harakiri = 60
listen = 10240
Notes:
chdir = /home/web/app - it's the path to your application
module = core.wsgi - your application should have directory with main (core) application called core (you should see wsgi.py there)
pidfile=/tmp/app.pid - just a place for pid file
socket = 0.0.0.0:8080 - it will listen port 8080
env = DJANGO_SETTINGS_MODULE=core.settings - again, you need main app to be called core (it should be inside core directory) and you should have settings.py inside it)
Docker / Docker Compose
You might need Docker and Docker Compose to orchestrate all that software. But it's possible to try run everything without it as well.
docker-compose.yml
version: "2.4"
services:
uwsgi:
build:
context: ./docker
dockerfile: Dockerfile
hostname: uwsgi
sysctls:
net.core.somaxconn: 10240
environment:
- DJANGO_SETTINGS_MODULE=${DJANGO_SETTINGS_MODULE}
- C_FORCE_ROOT=${C_FORCE_ROOT}
volumes:
- ./config/uwsgi/uwsgi.conf:/uwsgi.conf
- ../app:/home/web/app
- ./static:/home/web/static:rw
- ./media:/home/web/media:rw
- ./logs:/var/log/
restart: always
networks:
myapp_backend:
aliases:
- uwsgi
web:
image: nginx
hostname: nginx
sysctls:
net.core.somaxconn: 10240
depends_on:
- uwsgi
volumes:
# - ./config/nginx/nginx.conf:/etc/nginx/nginx.conf
- ./config/nginx/sites-enabled:/etc/nginx/conf.d
- ./media:/home/web/media:ro
- ./static:/home/web/static:ro
- ./logs:/var/log/nginx
ports:
- "80:80"
# - "443:443"
restart: always
networks:
- myapp_backend
networks:
myapp_backend:
Dockerfile
FROM python:3.9.0
RUN export DEBIAN_FRONTEND=noninteractive
ENV DEBIAN_FRONTEND noninteractive
RUN dpkg-divert --local --rename --add /sbin/initctl
RUN apt-get install -y --fix-missing && apt-get update -y
RUN apt-get install -y python3-pip \
python3-setuptools
COPY requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
RUN pip install uwsgi
WORKDIR /home/web/app
EXPOSE 8080
CMD ["uwsgi", "--ini", "/uwsgi.conf"]
Directory Structure
You can use directory structure from this repository: https://github.com/ansysy24/GameEconomy/tree/master/deployment
Also I highly recommend this article https://andrey-borzenko.medium.com/simple-nginx-uwsgi-daphne-reactive-application-part-1-the-big-picture-20d7b9ee5b96
And don't forget to add .env file like this one https://github.com/ansysy24/GameEconomy/blob/master/deployment/.env_template (but you should rename ofc)