Getting error 502 when processing an excel file with many rows - django

Question

Getting error 502 when processing an excel file with many rows.
Using Django / Nginx
The problem is not the weight of the file is less than 1Mb.
This page works correctly with files of 200 rows, the problem starts when the file have more rows and then, the page take too long processing this file.
This is the error:
2012/07/28 14:29:54 [error] 18515#0: *34 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: localhost, request: "POST /import/ HTTP/1.1", upstream: "http://127.0.0.1:9000/import/", host: "localhost:8080", referrer: "http://localhost:8080/import/"
I am using very large values ​​for the variables, but I keep getting the same error.
This is the configuration of the site:
upstream app_server {
server 127.0.0.1:9000 fail_timeout=3600s;
keepalive 3600s;
}
server {
listen 8080;
client_max_body_size 4G;
server_name localhost;
keepalive_timeout 3600s;
client_header_timeout 3600s;
client_body_timeout 3600s;
send_timeout 3600s;
location /static/ {
root /my path/;
autoindex on;
expires 7d;
}
location /media/ {
root /my path/;
autoindex on;
expires 7d;
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_connect_timeout 3600s;
proxy_send_timeout 3600s;
proxy_read_timeout 3600s;
if (!-f $request_filename) {
proxy_pass http://app_server;
break;
}
}
}
And this is the global configuration:
user www-data;
worker_processes 1;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
access_log /var/log/nginx/access.log;
sendfile on;
keepalive_timeout 3600s;
tcp_nodelay on;
client_header_timeout 3600s;
client_body_timeout 3600s;
send_timeout 3600s;
proxy_connect_timeout 3600s;
proxy_send_timeout 3600s;
proxy_read_timeout 3600s;
client_max_body_size 200m;
client_body_buffer_size 128k;
gzip on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Can you give me some help?
Best regards

Answer 1

The best option is rewrite the routine to use django-celery but if you want a quick solution you could try upgrading the timeout for your proxy pass in Nginx by adding:
proxy_connect_timeout 300s;
proxy_read_timeout 300s;
You should add this config on /var/nginx/sites-available/[site-config] to a specific site or /var/nginx/nginx.conf if you want to increase the timeout on all sites served by nginx.
If you are using gunicorn, You must add --timeout=300 as well. Example:
gunicorn_django -D -b 127.0.0.1:8901 --workers=2 --pid=/var/webapp/campus.pid --settings=settings.production --timeout 300 --pythonpath=/var/webapp/campus/
References:
http://wiki.nginx.org/HttpProxyModule#proxy_connect_timeout
http://reinout.vanrees.org/weblog/2011/11/24/apache-nginx-gunicorn-timeout.html
Gunicorn Nginx timeout problem