Django + Nginx save uploded file to other server and reverse file - django

I have 2 servers: first for Nginx and Django and second server for storage
Nginx and app server IP: 192.168.1.1
storage server IP: 192.168.1.2
Nginx installed on two servers.
In Django config media path:
MEDIA_ROOT = os.path.join(BASE_DIR,'media')
MEDIA_URL = '/media/'
Nginx config is in the first server:
server {
listen 80;
server_name 192.168.1.1;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log ;
location = /favicon.ico { access_log off; log_not_found off; }
location /media/ {
proxy_pass http://192.168.1.2/;
}
location / {
uwsgi_pass unix:/tmp/uwsgi/app.sock;
include uwsgi_params;
}
location /static/ {
alias /home/ubuntu/app/static/;
}
}
and storage server Nginx config:
server {
listen 80;
server_name 192.168.1.2;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log ;
location /media/ {
alias /home/ubuntu/app/media;
}
}
Questions:
How can Django save (upload file) to the storage server (192.168.1.2)?
better if suggest solutions with minimum changes in code.
How Nginx can reverse files from a storage server?
the end user just typing 192.168.1.1

Solution 1: Network File System (NFS)
An example of NFS is GlusterFS.
What it does is, it makes multiple disks or servers available as a single directory. So you can configure it to show your other server as media directory and any files you put in there, it will be automatically stored in your other server. And you can fetch those files as if they are in the media directory even though they are on another machine. No changes required to your django code, or nginx config.
Solution 2: Custom File Storage backend
Another solution is to write your own File Storage backend and save the images to your second server from there. In fact, there's a library called django-storages which supports uploading files to another server using FTP. See docs: http://django-storages.readthedocs.io/en/latest/backends/ftp.html
Personally, second solution seems better to me because you don't really need NFS right now. And even if you do later on, you can install Gluster on your second server and scale out from there.

Related

Nginx 504 Timeout Error within Docker container running Django and loading Scikit-learn model into memory

I'm building a web app with Django that uses a pre-trained scikit-learn model to process data that an user inputs through a web form. During development I'm able to load the model into memory by running the following command in urls.py
modelRF = joblib.load('model.pkl')
However, when I try to deploy the app inside a Docker container I receive a 504 Gateway Timeout Error. I've tried increasing the timeout limits in the nginx.conf file without any success. I was wondering whether this could also be a problem with the amount of memory assigned to the container.
I'm not sure whether the problem is related to Docker or to the way I'm loading the model into memory while in deployment (rather than in development). I'm using docker-compose with nginx, supervisor and uwsgi.
My nginx.conf file looks like this:
upstream django {
server unix:///tmp/uwsgi.sock; # for a file socket
}
server {
listen 80 default_server;
server_name .example.com;
charset utf-8;
# max upload size
client_max_body_size 75M;
# Django media
location /media {
alias /home/docker/code/media;
}
location /static {
alias /home/docker/code/static;
}
location / {
uwsgi_pass django;
include /home/docker/code/uwsgi_params;
}
}
Add uwsgi_read_timeout directive inside django location curly braces like this:
location / {
uwsgi_pass django;
include /home/docker/code/uwsgi_params;
uwsgi_read_timeout 3000;
}

Defining correctly Nginx server block for two django apps and no server_name

I have been following the Digital Ocean tutorial How To Serve Django Applications with uWSGI and Nginx on Ubuntu 14.04, so that later i can deploy my own django application using Nginx+uWSGI.
In this tutorial they create 2 basic Django apps to be later served by Nginx. I have tested that the apps were working using the Django server and uWSGI alone.
When i passed to the Nignx part i ran into a problem, basically i dont have a server_name for now only have an IP to work with, and i tried to differentiate between Django apps using the port number.
The default Nginx server (xxx.xxx.xxx.xxx:80) is responding correctly, but when i try to access the Django apps using (xxx.xxx.xxx.xxx:8080 or xxx.xxx.xxx.xxx:8081) i get 502 bad gateway.
I think i have a problem in the way or logic i am defining my listen inside the server block. What would be the correct way of doing this, or what might i be doing incorrectly.
This are my server blocks (in sites-enabled):
firstsite app
server {
listen xxx.xxx.xxx.xxx:8080;
#server_name _;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /root/firstsite;
}
location / {
include uwsgi_params;
uwsgi_pass unix:/root/firstsite/firstsite.sock;
}
}
econdsite app
server {
listen xxx.xxx.xxx.xxx:8081;
#server_name _;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /root/secondsite;
}
location / {
include uwsgi_params;
uwsgi_pass unix:/root/secondsite/secondsite.sock;
}
}
default Nginx
server {
listen 80 default_server;
#listen [::]:80 default_server ipv6only=on;
root /usr/share/nginx/html;
index index.html index.htm;
# Make site accessible from http://localhost/
server_name localhost;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
# Uncomment to enable naxsi on this location
# include /etc/nginx/naxsi.rules
}
}
UPDATE:
I was checking the error log under /var/log/nginx and when i try to connect to firstsite i get the following error:
2016/02/05 15:55:23 [crit] 11451#0: *6 connect() to
unix:/root/firstsite/firstsite.sock failed (13: Permission denied)
while connecting to upstream, client: 188.37.180.101, server: ,
request: "GET / HTTP/1.1", upstream:
"uwsgi://unix:/root/firstsite/firstsite.sock:", host:
"178.62.229.183:8080"
Nginx server on ubuntu will run on www-data user by default, uWSGI server won't (which is actually a good thing, unless it runs on root). If you're creating unix socket for uWSGI, access to it will be defined as for any system file. And by default, access to it might be restricted only to user that created socket.
More on that, you're creating your sockets in /root/ directory. That directory is readable only by root user and some of Linux distributions won't allow accessing anything inside even if permissions are set correctly.
So what you have to do is:
put sockets outside of /root/ directory (/var/run is good place for that)
Make sure that nginx will have access to that sockets (put --chmod-socket 666 or `--chown-socket yourusername:www-data into your uWSGI startup line)
And if you're running that uWSGI server on root, be aware that this is really dangerous. Any process running on root can do anything with your system, so if you will make mistake in your code or someone will hack in, he can inject any malicious software into your server, steal some data from it or just destroy everything.

Django Nginx Browser Caching Configuration

I am trying to configure Nginx to leverage on static file caching on browser.
My configuration file is as following
server {
listen 80;
server_name localhost;
client_max_body_size 4G;
access_log /home/user/webapps/app_env/logs/nginx-access.log;
error_log /home/user/webapps/app_env/logs/nginx-error.log;
location /static/ {
alias /home/user/webapps/app_env/static/;
}
location /media/ {
alias /home/user/webapps/app_env/media/;
}
...
}
When I add in the following caching configuration, the server fails to load static files and I am not able to restart my Nginx.
location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
expires 365d;
}
The nginx-error log shows open() "/usr/share/nginx/html/media/cover_photos/292f109e-17ef-4d23-b0b5-bddc80708d19_t‌​humbnail.jpeg" failed (2: No such file or directory)
I have done quite some research online but cannot solve this problem.
Can anyone help me or just give me some suggestions on implementing static file caching in Nginx?
Thank you!
Reference: Leverage browser caching for Nginx
Again, I have to answer my own question.
The root problem lays on the "path".
I find the answer from #Dayo, here I quote:
You are missing the rootdirective for the images location block.
Therefore, nginx will look for the files in the default location which
varies by installation and since you have most likely not placed the
files there, you will get a 404 Not Found error.
Answer from Dayo
Thus, I added the root path in my configuration file as following:
root /home/user/webapps/app_env/;
The whole configuration will look like this:
server {
listen 80;
server_name localhost;
root /home/user/webapps/app_env/;
client_max_body_size 4G;
access_log /home/user/webapps/app_env/logs/nginx-access.log;
error_log /home/user/webapps/app_env/logs/nginx-error.log;
location /static/ {
alias /home/user/webapps/app_env/static/;
}
location /media/ {
alias /home/user/webapps/app_env/media/;
}
location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
expires 365d;
}
...
}
And everything just work nice.
I hope people with the same problem can learn from this.

django+nginx+gunicorn - subdomain joy

i'm trying to setup django on nginx + gunicorn on a centos6 server (firewall off, selinux disabled). the project works locally on the server (tested running gunicorn on 127.0.0.1:8221), but not on the whole network. the project should be accessible from a subdomain project.mydomain.com
the project itself is located on a server centos6.mydomain.com and the dns server is main.mydomain.com
my ngnix conf for the project:
upstream project {
server 127.0.0.1:8221 fail_timeout=0;
}
server {
listen 80;
server_name project.mydomain.com;
access_log /var/log/nginx/project.mydomain.com.log;
error_log /var/log/nginx/project.mydomain.com.log;
root /home/USER/djangosites/project;
location / {
proxy_set_header Host $host;
if (!-f $request_filename){
proxy_pass http://project;
break;
}
}
location /media {
alias /home/USER/djangosites/project/media;
}
location /static {
alias /home/USER/djangosites/project/static;
}
}
nginx conf for the centos6 (working)
server {
listen 80 default_server;
server_name centos6.mydomain.com;
access_log /var/log/nginx/centos6.mydomain.com.access.log main;
error_log /var/log/nginx/centos6.mydomain.com.error.log;
location / {
root /var/www/centos6.mydomain.com;
index index.html;
}
}
gunicorn conf
import multiprocessing
bind = "127.0.0.1:8221"
logfile = "/home/USER/djangosites/project/gunicorn.log"
workers = multiprocessing.cpu_count() * 2 + 1
would i be better off giving a new ip (to the outside) to the project that is different from centos6.mydomain.com or i can just use the same ip with different local port?
how should i configure hosts.db on main.mydomain.com then?
centos6 A xxx.xxx.xxx.220
project A xxx.xxx.xxx.221
or
centos6 A xxx.xxx.xxx.220
project A xxx.xxx.xxx.220:8221
or
centos6 A xxx.xxx.xxx.220
project CNAME centos6
i'm kind of more inclined to give a new ip because everything is behind a m0n0wall, so a new ip could perhaps be easier to manage.
so basically, i'm guessing that my nginx conf for the project is flawed. what should i do with it?
ok. got it working :)
hosts.db on main.mydomain.com
project CNAME centos6
gunicorn runnig on 127.0.0.1:8221
and edited the nginx conf as stated above.

Nginx , SImple configuration for serving all the files in a Directory and all the directories within

I am looking for a simple configuration to serve all files and directories inside a particular folder.
To be more precise I am trying to serve everything inside the pinax /static_media/ folder and /media/ folder as it is with the same url, and preferably auto index everything .
by the way I have run python manage.py build_media --all so all static content is under <project_name>/site_media/static
The current configuration I am using :
server {
listen 80;
server_name QuadraPaper;
access_log /home/gdev/Projects/QuardaPaper/access_log.log;
location ^*/site_media/*$
{
autoindex on;
access_log off;
root /home/gdev/Projects/QuardaPaper/site_media;
}
location /media/ {
autoindex on;
root /home/gdev/Projects/QuardaPaper/media/;
}
All the different configuration instructions from various sites has really confused me , for example
How to serve all existing static files directly with NGINX, but proxy the rest to a backend server.
http://coffeecode.net/archives/200-Using-nginx-to-serve-static-content-with-Evergreen.html
https://serverfault.com/q/46315/91723
http://wiki.nginx.org/Pitfalls
http://pinaxproject.com/docs/0.7/media/#ref-media-devel
Environment information:
Xubuntu 10.04 running on VirtualBox
nginx 1.1.4
pinax 0.72
django 1.0.4
fastcgi for running django via nginx
I found the answer , It was Quite simple as i guessed .
One has to set the root directory once and use the sub-directories as the location
server {
listen 80;
server_name QuadraPaper;
access_log /home/gdev/Projects/QuardaPaper/access_log.log;
root /home/gdev/Projects/QuardaPaper;
location /site_media/ {
autoindex on;
access_log off;
}
location /media/ {
autoindex on;
}
}
I got a clue from
Nginx doesn't serve static