Slow uploads Nginx and Gluster - django

We are having trouble with uploads to my site with django, gunicorn, running behind nginx. we also have a gluster mount on the app server where the files are uploaded and distributed-replicated across several servers. (All tiers are on AWS)
When we go to upload a file(~15mb), we get a 502 Bad Gateway. we also check the nginx logs which show a upstream prematurely closed connection while reading response header from upstream, client. Our upload speeds are being extremely slow (<5k). we can upload to other sites just fine, and our internet upload is around 10MB with anything else.
Is there any configuration file that I am missing to allow uploads of a file through gunicorn or nginx?
nginx.conf
user www-data;
worker_processes 4;
pid /run/nginx.pid;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
server_names_hash_bucket_size 256;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/x-javascript text/xml
application/xml application/xml+rss text/javascript;
##
# nginx-naxsi config
##
# Uncomment it if you installed nginx-naxsi
##
#include /etc/nginx/naxsi_core.rules;
##
# nginx-passenger config
##
# Uncomment it if you installed nginx-passenger
##
#passenger_root /usr;
#passenger_ruby /usr/bin/ruby;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
conf.d files:
client_max_body_size 256m;
_
proxy_read_timeout 10m;
proxy_buffering off;
send_timeout 5m;
_
We have a feeling that it may be either nginx or the gluster mount. We have been working on this for days, and have looked all through the timeout* variables in nginx and gunicorn and haven't made any progress.
Any help would be appreciated, Thank you!

So, we solved the problem. It had nothing to do with any of our code, server setup, or amazon. We narrowed it down to only linux machines uploading in our network. there was a bug with 'tcp window scaling' in the firewall that was resetting the upload after it reaches a limit.
Thanks for anyone that attempted.

Related

Nginx status is active but it is not serving

I have attached two files in which I am hosting var/www/html files and localhost/:3000
please help me why nginx not serving when i am hitting ip of server.
Is there any solution for my problem if so then let me know what changes should I do so that it will work
I have configure port 81 for this application
nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml>
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
sites-enabled file
server {
listen 81 default_server;
listen [::]:81 default_server;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.php;
server_name _;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ /index.php?args;
}
location /front/ {
proxy_pass http://localhost:3000/;
}
# pass PHP scripts to FastCGI server
#
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
# # With php-cgi (or other tcp sockets):
# fastcgi_pass 127.0.0.1:9000;
}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
location ~ /\.ht {
deny all;
}
}
In your NGINX config you've set port 81 but you're trying to hit port 3000?
Other than that verify
If the files working within the server? Like CURL or WGET
Make sure you've configured the security group to open the correct port for incoming traffic
Make sure that you're using PUBLIC IP of your instance
Make sure that your instance have access to server (should be in public subnet)
Even after all this if it doesn't work then update the question with more details as to exact error message.

GUnicorn and Django for 500 concurrent requests with limited resources

I was tasked with creating a Django-Gunicorn demo app. In this task, I need to be able to handle 500 concurrent login requests in 1 second.
I have to deploy the app in a VM with 2GB RAM and 2 core CPUs (using Vagrant and VirtualBox, Ubuntu 16.04). I already tried the following for deployment.
gunicorn --workers 5 --bind "0.0.0.0:8000" --worker-class "gevent" --keep-alive 5 project.wsgi
Using JMeter test from the host machine, the test always takes around 7-10 seconds. Even if the login endpoint only returns empty response without any database access, the amount of the time is almost the same. Can you tell me what's wrong with this?
I use the default settings at /etc/nginx/nginx.conf.
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 768;
multi_accept on;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
And here is my reverse proxy settings I put in sites-available folder.
server {
listen 80;
location /static {
autoindex on;
alias /vagrant/static/;
}
location /media {
autoindex on;
alias /vagrant/uploads/;
}
location / {
proxy_redirect http://127.0.0.1:8000/ http://127.0.0.1:8080/;
proxy_pass http://127.0.0.1:8000;
}
}
Thanks
The short answer is that you miss worker connections in gunicorn. So it cannot handle more concurrent requests.
For 500 concurrent login requests, the concurrent active connections handled by the database is also important. If the database cannot handle the loading, you are going to fail too. If you're using PSQL, you have to change max connections and use a connection pool

Nginx still displaying welcome page even after deleting default from sites-enabled

I am new to nginx. i followed the following tutorial.
http://michal.karzynski.pl/blog/2013/06/09/django-nginx-gunicorn-virtualenv-supervisor/
The problem is that instead of displaying my site's homepage, nginx always displays its welcome page. I have deleted the default symbolic link from /etc/nginx/sites-enabled and restarted nginx but i still have not been able to get rid of the welcome page. Can anyone advise, what can I do to see my actual website?
EDIT:
here is the config
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
#mail {
# # See sample authentication script at:
# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
# # auth_http localhost/auth.php;
# # pop3_capabilities "TOP" "USER";
# # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
# server {
# listen localhost:110;
# protocol pop3;
# proxy on;
# }
#
# server {
# listen localhost:143;
# protocol imap;
# proxy on;
# }
#}
# configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;
text/mathml mml;
text/plain txt;
text/vnd.sun.j2me.app-descriptor jad;
text/vnd.wap.wml wml;
text/x-component htc;
image/png png;
image/tiff tif tiff;
image/vnd.wap.wbmp wbmp;
image/x-icon ico;
image/x-jng jng;
image/x-ms-bmp bmp;
image/svg+xml svg svgz;
image/webp webp;
application/font-woff woff;
application/java-archive jar war ear;
application/json json;
application/mac-binhex40 hqx;
application/msword doc;
application/pdf pdf;
application/postscript ps eps ai;
application/rtf rtf;
application/vnd.apple.mpegurl m3u8;
application/vnd.ms-excel xls;
application/vnd.ms-fontobject eot;
application/vnd.ms-powerpoint ppt;
application/vnd.wap.wmlc wmlc;
application/vnd.google-earth.kml+xml kml;
application/vnd.google-earth.kmz kmz;
application/x-7z-compressed 7z;
application/x-cocoa cco;
application/x-java-archive-diff jardiff;
application/x-java-jnlp-file jnlp;
application/x-makeself run;
application/x-perl pl pm;
application/x-pilot prc pdb;
application/x-rar-compressed rar;
application/x-redhat-package-manager rpm;
application/x-sea sea;
application/x-shockwave-flash swf;
application/x-stuffit sit;
application/x-tcl tcl tk;
application/x-x509-ca-cert der pem crt;
application/x-xpinstall xpi;
application/xhtml+xml xhtml;
application/xspf+xml xspf;
application/zip zip;
application/octet-stream bin exe dll;
application/octet-stream deb;
application/octet-stream dmg;
application/octet-stream iso img;
application/octet-stream msi msp msm;
application/vnd.openxmlformats-officedocument.wordprocessingml.document docx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet xlsx;
application/vnd.openxmlformats-officedocument.presentationml.presentation pptx;
audio/midi mid midi kar;
audio/mpeg mp3;
audio/ogg ogg;
audio/x-m4a m4a;
audio/x-realaudio ra;
video/3gpp 3gpp 3gp;
video/mp2t ts;
video/mp4 mp4;
video/mpeg mpeg mpg;
video/quicktime mov;
video/webm webm;
video/x-flv flv;
video/x-m4v m4v;
video/x-mng mng;
video/x-ms-asf asx asf;
video/x-ms-wmv wmv;
video/x-msvideo avi;
}
# configuration file /etc/nginx/sites-enabled/Preonlab_webapp:
upstream Preonlab_wepapp_server {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:[myserverpath]/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name localhost;
client_max_body_size 4G;
access_log [myserverpath]/logs/nginx-access.log;
error_log [myserverpath]/logs/nginx-error.log;
location / {
# an HTTP header important enough to have its own Wikipedia entry:
# http://en.wikipedia.org/wiki/X-Forwarded-For
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# enable this if and only if you use HTTPS, this helps Rack
# set the proper protocol for doing redirects:
# proxy_set_header X-Forwarded-Proto https;
# pass the Host: header from the client right along so redirects
# can be set properly within the Rack application
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
# set "proxy_buffering off" *only* for Rainbows! when doing
# Comet/long-poll stuff. It's also safe to set if you're
# using only serving fast clients with Unicorn + nginx.
# Otherwise you _want_ nginx to buffer responses to slow
# clients, really.
# proxy_buffering off;
}
}

Rails 4 + Unable to upload the APK file at AWS Server using NGINX

In my Rails App, I am able to upload the APK file locally and storage place is S3. But the same stuff I tried from AWS server by making the code live. Unable to upload the file.
For Same record, thumbnail and screenshot can be uploaded successfully, but for APK file, it executes the whole process but after page reload, it does not store the record.
I am thinking, some thing is wrong with NGINX setting.
NGINX config file :-
user www-data;
worker_processes 4;
pid /run/nginx.pid;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 600s;
send_timeout 600s;
types_hash_max_size 2048;
client_max_body_size 300M;
client_body_in_file_only clean;
client_body_buffer_size 32K;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Phusion Passenger config
##
# Uncomment it if you installed passenger or passenger-enterprise
##
passenger_root /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini;
passenger_ruby /usr/bin/passenger_free_ruby;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Not getting the way to resolve it.

nginx/uwsgi don't serve any request

I have nginx, uwsgi, django stack. There use to be an https serving from the nginx but I'm trying to disable it.
this is my nginx configuration files:
nginx.conf:
events {
worker_connections 768;
# multi_accept on;
}
http {
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
##
# nginx-naxsi config
##
# Uncomment it if you installed nginx-naxsi
##
#include /etc/nginx/naxsi_core.rules;
##
# nginx-passenger config
##
# Uncomment it if you installed nginx-passenger
##
#passenger_root /usr;
#passenger_ruby /usr/bin/ruby;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
site-available/mysite.com:
upstream django {
server 127.0.0.1:44000;
}
server {
listen 80 default;
server_name www.mysite.com mysite.com;
charset utf-8;
access_log /var/log/nginx/mysite_access.log;
error_log /var/log/nginx/mysite_error.log;
location /static/ {
root /opt/myapp/myapp/;
index index.html index.htm;
}
location / {
uwsgi_pass django;
include uwsgi_params;
}
}
When I run it with uwsgi i get this error:
django.core.exceptions.SuspiciousOperation: Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS): www.mysite.com
[pid: 12865|app: 0|req: 1/2] 84.94.36.146 () {36 vars in 571 bytes} [Mon May 26 07:12:55 2014] GET /favicon.ico => generated 0 bytes in 457 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
I put 'www.mysite.com' on my ALLOWED_HOSTS section in settings.py and it didn't help.
these are the lines I added in settings.py:
ALLOWED_HOSTS = [
'.mysite.com', # Allow domain and subdomains
'.mysite.com.', # Also allow FQDN and subdomains
]
When I run it with runserver i get this error:
[26/May/2014 06:53:09] code 400, message Bad request syntax ('\x00;\x02\x00\x0c\x00QUERY_STRING\x00\x00\x0e\x00REQUEST_METHOD\x03\x00GET\x0c\x00CONTENT_TYPE\x00\x00\x0e\x00CONTENT_LENGTH\x00\x00\x0b\x00REQUEST_URI\x0c\x00/favicon.ico\t\x00PATH_INFO\x0c\x00/favicon.ico\r\x00DOCUMENT_ROOT\x0f\x00/etc/nginx/html\x0f\x00SERVER_PROTOCOL\x08\x00HTTP/1.1\x0b\x00REMOTE_ADDR\x0c\x0084.94.36.146\x0b\x00REMOTE_PORT\x05\x0051792\x0b\x00SERVER_PORT\x02\x0080\x0b\x00SERVER_NAME\x10\x00www.mysite.com\t\x00HTTP_HOST\x10\x00www.mysite.com\x0f\x00HTTP_CONNECTION')
[26/May/2014 06:53:09] ";
QUERY_STRINGREQUEST_METHODGET
CONTENT_TYPECONTENT_LENGTH
REQUEST_URI
/favicon.ico PATH_INFO
DOCUMENT_ROOT/etc/nginx/htmlSERVER_PROTOCOHTTP/1.1 /favicon.ico
REMOTE_ADDR
84.94.36.146
REMOTE_PORT51792
SERVER_PORT80
SERVER_NAMEwww.mysite.com HTTP_HOSTwww.mysite.comHTTP_CONNECTION" 400 -
I read that this error may be caused by HTTPS request, but it doesn't seem to me that I redirect HTTP requests to HTTPS anywhere, cause I deleted all those configurations.
any ideas?
Thanks
It's working now.
The problem was that the ALLOWED_HOSTS was already defined in the local_settings.py with the old domains. I updated the domains and it worked.
All the nginx configurations are correct, so the files attached here are useful.
Thanks for your time.
I solved it by using
uwsgi --socket :8001 --wsgi-file test.py
instead of
python manage.py runserser