nginx file upload freezes - django

Question

I'm testing a django website deployment. The site works without any issues when I connect directly to my gunicorn localhost and run it in debug mode (so that django handles file uploads itself). When I access the site with debug mode turned off through nginx (it binds to the same gunicorn localhost) everything works just as well, except file uploads. Whenever I try to upload a file > 1MB, the upload freezes at some point (with a 1.3MB file, my browser freezes at 70%).
I've installed nginx into a conda virtual environment (conda install --no-update-dependencies -c anacoda nginx). Here is the etc/nginx.conf file:
# nginx Configuration File
# https://www.nginx.com/resources/wiki/start/topics/examples/full/
# http://nginx.org/en/docs/dirindex.html
# https://www.nginx.com/resources/wiki/start/
# Run as a unique, less privileged user for security.
# user nginx www-data; ## Default: nobody
# If using supervisord init system, do not run in deamon mode.
# Bear in mind that non-stop upgrade is not an option with "daemon off".
# daemon off;
# Sets the worker threads to the number of CPU cores available in the system
# for best performance.
# Should be > the number of CPU cores.
# Maximum number of connections = worker_processes * worker_connections
worker_processes auto; ## Default: 1
# Maximum number of open files per worker process.
# Should be > worker_connections.
# http://blog.martinfjordvald.com/2011/04/optimizing-nginx-for-high-traffic-loads/
# http://stackoverflow.com/a/8217856/2127762
# Each connection needs a filehandle (or 2 if you are proxying).
worker_rlimit_nofile 8192;
events {
# If you need more connections than this, you start optimizing your OS.
# That's probably the point at which you hire people who are smarter than
# you as this is *a lot* of requests.
# Should be < worker_rlimit_nofile.
worker_connections 8000;
}
# Log errors and warnings to this file
# This is only used when you don't override it on a server{} level
#error_log logs/error.log notice;
#error_log logs/error.log info;
error_log var/log/nginx/error.log warn;
# The file storing the process ID of the main process
pid var/run/nginx.pid;
http {
# Log access to this file
# This is only used when you don't override it on a server{} level
access_log var/log/nginx/access.log;
# Hide nginx version information.
server_tokens off;
# Controls the maximum length of a virtual host entry (ie the length
# of the domain name).
server_names_hash_bucket_size 64;
# Specify MIME types for files.
include mime.types;
default_type application/octet-stream;
# How long to allow each connection to stay idle.
# Longer values are better for each individual client, particularly for SSL,
# but means that worker connections are tied up longer.
keepalive_timeout 20s;
# Speed up file transfers by using sendfile() to copy directly
# between descriptors rather than using read()/write().
# For performance reasons, on FreeBSD systems w/ ZFS
# this option should be disabled as ZFS's ARC caches
# frequently used files in RAM by default.
sendfile on;
# Don't send out partial frames; this increases throughput
# since TCP frames are filled up before being sent out.
tcp_nopush on;
# Enable gzip compression.
gzip on;
# Compression level (1-9).
# 5 is a perfect compromise between size and CPU usage, offering about
# 75% reduction for most ASCII files (almost identical to level 9).
gzip_comp_level 5;
# Don't compress anything that's already small and unlikely to shrink much
# if at all (the default is 20 bytes, which is bad as that usually leads to
# larger files after gzipping).
gzip_min_length 256;
# Compress data even for clients that are connecting to us via proxies,
# identified by the "Via" header (required for CloudFront).
gzip_proxied any;
# Tell proxies to cache both the gzipped and regular version of a resource
# whenever the client's Accept-Encoding capabilities header varies;
# Avoids the issue where a non-gzip capable client (which is extremely rare
# today) would display gibberish if their proxy gave them the gzipped version.
gzip_vary on;
# Compress all output labeled with one of the following MIME-types.
gzip_types
application/atom+xml
application/javascript
application/json
application/ld+json
application/manifest+json
application/rss+xml
application/vnd.geo+json
application/vnd.ms-fontobject
application/x-font-ttf
application/x-web-app-manifest+json
application/xhtml+xml
application/xml
font/opentype
image/bmp
image/svg+xml
image/x-icon
text/cache-manifest
text/css
text/plain
text/vcard
text/vnd.rim.location.xloc
text/vtt
text/x-component
text/x-cross-domain-policy;
# text/html is always compressed by gzip module
# This should be turned on if you are going to have pre-compressed copies (.gz) of
# static files available. If not it should be left off as it will cause extra I/O
# for the check. It is best if you enable this in a location{} block for
# a specific directory, or on an individual server{} level.
# gzip_static on;
include conf.d/*.conf;
}
This is the original version of my server's configuration file (conf.d/test.conf).
server {
server_name localhost;
listen 8081;
access_log on;
client_max_body_size 32M;
send_timeout 100s;
location /static/ {
alias /Users/user/static/;
autoindex on;
error_log /Users/user/.nginx/labsite.static.error.log warn;
}
location /media/ {
alias /Users/user/media/;
autoindex on;
error_log /Users/user/.nginx/labsite.media.error.log warn;
}
location / {
proxy_pass http://localhost:8001;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Real-IP $remote_addr;
add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
}
access_log /Users/user/.nginx/labsite.access.log combined;
error_log /Users/user/.nginx/labsit.error.log warn;
}
I've found several related posts:
Nginx PHP Failing with Large File Uploads (Over 6 GB)
https://serverfault.com/questions/626817/nginx-file-upload-pauses-stalls-in-the-middle-uploads-only-258kb-and-stops
https://easyengine.io/tutorials/php/increase-file-upload-size-limit/
They led me to introduce some modifications
server {
server_name localhost;
listen 8081;
access_log on;
client_max_body_size 32M;
send_timeout 300s;
gzip_static off;
location /static/ {
alias /Users/user/static/;
autoindex on;
error_log /Users/user/.nginx/labsite.static.error.log warn;
}
location /media/ {
alias /Users/user/media/;
client_body_temp_path /Users/user/media;
autoindex on;
error_log /Users/user/.nginx/labsite.media.error.log warn;
}
location / {
proxy_pass http://localhost:8001;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Real-IP $remote_addr;
add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
}
access_log /Users/user/.nginx/labsite.access.log combined;
error_log /Users/user/.nginx/labsit.error.log warn;
}
I've also tried setting sendfile off in my config file, because that's recommended for Free BSD (and Mac OS X is based on Free BSD), but to no avail. Am I missing something?

Answer 1

Seems like, I've figured this out. I had to change the temporary directory (I'm not entirely sure why, because there were no permissions-related issues) and set/increase the client_body_timeout parameter.
server {
listen 8081;
server_name localhost;
client_max_body_size 32M;
client_body_timeout 300s;
send_timeout 300s;
client_body_temp_path /Users/user/media;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /Users/user;
}
location /media/ {
root /Users/user;
}
location / {
proxy_pass http://localhost:8001;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Real-IP $remote_addr;
add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
}
}