Configure Nginx and Django for Byte Range Request - django

I am building a web API using Django and Nginx that needs to support Byte-Range requests.
Currently, when making a request such as:
curl --header "Content-Type: application/json"
--header "Authorization: Token 8f25cd3bb5b43a5277f4237b1d1db0273dbe8f6a"
--request POST http://myurl/download/
--header "Range: bytes=0-50"
-v
my Django view (the MyFileAPI class) is called, the user is authenticated, and the correct file path (/code/web/myapp/user_id/user_info.pbf) is fetched. However, the entire file is returned instead of only the first 50 bytes.
How do I configure Nginx and Django to allow partial file downloads?
default.conf
server {
listen 80;
server_name localhost;
charset utf-8;
client_max_body_size 10M;
add_header Accept-Ranges bytes;
proxy_force_ranges on;
location /static/ {
alias /django_static/;
}
location / {
include uwsgi_params;
uwsgi_pass web:8000;
}
}
nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
views.py
class MyFileAPI(APIView):
permission_classes = (IsAuthenticated,)
def post(self, request):
try:
user = request.user
my_file = MyFile.objects.filter(user_id=user).latest("created_date")
# ex) my_file.filepath = /myapp/13/user_info.pbf
# ex) my_files_path = /code/web/myapp/13/user_info.pbf
my_files_path = settings.BASE_DIR + my_file.filepath
response = FileResponse(open(my_files_path, 'rb'))
response['Accept-Ranges'] = 'bytes'
response['X-Accel-Buffering'] = 'no'
# Adding this next line results in a 404 error
# response['X-Accel-Redirect'] = my_files_path
return response
except Exception as e:
return Response({'status': str(e)}, content_type="application/json")
When I add the line response['X-Accel-Redirect'] = my_files_path I receive the error The current path /myapp/13/user_info.pbf didn't match any of these.
and all of my urls from urls.py are listed.
I know there are other posts on the same topic, but none give a full answer.

Related

Bad Request on File upload

I have setup my python project using nginx and gunicorn as described in this tutorial https://djangocentral.com/deploy-django-with-nginx-gunicorn-postgresql-and-lets-encrypt-ssl-on-ubuntu/
so far everything works perfectly fine.
My project is in /root/project_folder/
I want to upload files via the admin page, but I get a bad request error 400. The media folder I want to add the files to is /var/www/my_domain/media (owned by the group www-data). This is also properly configured since I can see the images when I move them manually into that folder.
Do you guys maybe have any idea why the issue may be ?
the main problem is when uploading I need to save the image to the media root:
/var/www/domain/media/images/dad/ProfilePicture/offline_screen.png
But when I send request to view the image the web server returns the following:
/media/var/www/domain/media/images/dad/ProfilePicture/offline_screen.png
The upload_to path needs to be images/dad/ProfilePicture/offline_screen.png for correct request but then the image is uploaded to the wrong folder
Any ideas ? Thanks in advance!
UPDATE:
Here is my nginx config
`
server {
server_name domain.net www.domain.net ip_address;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /var/www/domain;
}
location = /media/ {
root /var/www/domain;
}
location / {
include proxy_params;
proxy_pass http://unix:/var/log/gunicorn/domain.sock;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/domain/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/domain/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = www.domain) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = domain) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name domain www.domain ip_address;
return 404; # managed by Certbot
}
`
Here is my media root / media url:
MEDIA_ROOT = '/var/www/domain/media' #os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'
The url.py:
urlpatterns = [
path('admin/', admin.site.urls),
path('', include('webpages.urls')),
]
if settings.DEBUG:
urlpatterns += static(settings.MEDIA_URL,document_root=settings.MEDIA_ROOT)
urlpatterns += static(settings.STATIC_URL,document_root=settings.STATIC_ROOT)
And finally how I serve and upload Images:
def get_path_of_content_image(instance, filename):
return os.path.join(settings.MEDIA_ROOT, "images", str(instance.course.topic), str(instance.course), filename)
class Topic(models.Model):
title = models.CharField(max_length=100)
description = models.TextField(max_length=445)
image = models.ImageField(upload_to=get_topic_profile_path, default="images/default.jpeg")
def __str__(self):
return self.title
def image_tag(self):
if self.image:
return mark_safe('<img src="%s" style="width: 200px; height:200px;" />' % self.image.get_image())
def get_image(self):
if self.image:
return str(self.image.url).replace('/media/var/www/domain', '') ########1
else:
return settings.STATIC_ROOT + 'webpages/img/default.jpeg'
image_tag.short_description = 'Current Image'
I wrote this line #######1 since set the upload path to the full media root but then in order to return the correct path I need to remove the media root prefix. And also very importantly this only works if DEBUg set to TRUE.
I'm not certain this will solve your issue, but there are 2 things that need to be configured correctly here. If nginx is serving the files in your media directory ok, I'm guessing your nginx config is already ok and you just need to edit settings.py, but I'll include the nginx config in case it helps someone else.
UPDATE
I don't think those methods in your models are necessary, although I don't really know how they are used. Here's an example of how I configure a model with an ImageField:
Firstly, how to upload and save images
(letting django do most of the work)
models.py
...
class Post(models.Model):
"""Represents a single blog post"""
# model fields
title = models.CharField(max_length=150, unique=True)
content = models.TextField()
title_image = models.ImageField(upload_to='image/', blank=True, null=True)
...
forms.py
...
class PostModelForm(forms.ModelForm):
class Meta:
model = Post
fields = ['title', 'title_image', 'content']
widgets = {
'content': Textarea(attrs={'cols': 80, 'rows': 25}),
}
...
views.py
...
def new_post(request):
context = {}
template = 'new_post.html'
# POST
if request.method == 'POST':
form = PostModelForm(request.POST or None, request.FILES)
if form.is_valid():
new_post = form.save()
return redirect('list_post')
else:
context['form'] = form
return render(request, template, context)
# GET
else:
context['form'] = PostModelForm()
return render(request, template, context)
...
new_post.html (note the enctype of the form - important if you want to upload files)
...
<div class="card bg-light">
<div class="card-header">
<h3>New Post</h3>
</div>
<div class="card-body">
<form action="{% url 'blog:new_post' %}" method="POST" class="w-100" enctype="multipart/form-data">
<div class="col-12">
{% csrf_token %}
{{ form.as_p }}
</div>
<div class="col-12">
<ul class="actions">
<button class="primary" type="submit">Save</button>
</ul>
</div>
</form>
</div>
</div>
...
Secondly, how to serve your images
Now you want to display the image in you markup
In views.py
...
def detail(request, id):
context = {}
template_name = 'post.html'
post = get_object_or_404(Post, id=id)
context['post'] = post
return render(request, template_name, context)
...
In your markup in post.html
...
{% if post.title_image %}
<span class="img"><img src="{{ post.title_image.url }}" alt="image for {{ post.title }}" /></span>
{% endif %}
...
(1) Define MEDIA_ROOT and MEDIA_URL in settings.py
In settings.py add
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'
Note: Read the django docs on MEDIA_ROOT and MEDIA_URL
(2) define media location in nginx server block
In that tutorial the nginx configuration doesn't define a location for media. Here's the server block from the tutorial:
server {
listen 80;
server_name server_domain_or_IP;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /path/to/staticfiles;
}
location / {
include proxy_params;
proxy_pass http://unix:/var/log/gunicorn/project_name.sock;
}
}
I think it should look something like this:
server {
listen 80;
server_name server_domain_or_IP;
location = /favicon.ico { access_log off; log_not_found off; }
location /media/ {
root /path/to/media;
}
location /static/ {
root /path/to/staticfiles;
}
location / {
include proxy_params;
proxy_pass http://unix:/var/log/gunicorn/project_name.sock;
}
}
If that configuration doesn't work, I've posted what works for me below
In my own experience with django, nginx and gunicorn however, my nginx configuration looks like this:
server {
listen 443 ssl;
listen [::]:443 ssl;
ssl_certificate /etc/letsencrypt/live/website_name.com-0001/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/website_name.com-0001/privkey.pem; # managed by Certbot
server_name website_name.com www.website_name.com;
location = /favicon.ico { access_log off; log_not_found off; }
location /media {
alias /home/user_name/website_name/project_name/media;
}
location /static/ {
alias /home/user_name/website_name/project_name/static_root/;
}
location / {
include proxy_params;
proxy_pass http://unix:/run/gunicorn.sock;
}
}
Note 1: Here are the nginx docs for alias and root directives.
So after spending an eternity on debugging, I have the solution to my problem:
First Thing that was problematic is the '=' between 'location' and '/media/' in the nginx file. Be sure that this is not root of the error. It should look like this:
location /media {
alias /home/user_name/website_name/project_name/media;
}
and not like this:
location = /media {
alias /home/user_name/website_name/project_name/media;
}
The file upload to a custom local directory can be done like this:
from django.core.files.storage import FileSystemStorage
fs = FileSystemStorage(location=settings.MEDIA_ROOT)
and for the model add the file storage option on upload (or change the name of the image with the updated path afterwards):
image = models.ImageField(upload_to=get_topic_profile_path, default="images/default.jpeg", storage=fs)
thanks for the other answers. I learned quite a lot by fixing this erros :D

Nginx : How to serve multiple contents with a single internal directive after authentication

I want to display my documentation (a single-page application built with React) only after authentication with my backend.
My configuration :
Nginx acts as a reverse proxy for the backend (Django) and serves static files like single-page-applications.
Django, the backend, identifies the user and makes a request to Nginx using X-Accel-Redirect.
So I proceed as follows:
1) Authentication on Django
views.py
def get_doc(request):
if request.method == 'POST':
form = PasswordForm(request.POST)
if form.is_valid():
if form.cleaned_data['password'] == 'foo':
response = HttpResponse()
response['Content-Type'] = ''
response['X-Accel-Redirect'] = '/docs-auth/'
return response
else:
return HttpResponse("Wrong password")
else:
form = PasswordForm()
return render(request, 'docs/form.html', {'form': form})
urls.py
urlpatterns = [
path('docs/', views.get_doc, name='documentation'),
]
2) Nginx serves the single-page application
upstream backend {
server web:8000;
}
server {
location = /favicon.ico {access_log off;log_not_found off;}
...
location /docs {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
proxy_pass http://backend;
}
location /docs-auth/ {
internal;
alias /home/foo/docs/;
index index.html;
try_files $uri $uri/ /docs/index.html;
}
location / {
alias /home/foo/landing_page/;
error_page 404 /404.html;
index index.html;
try_files $uri $uri/ =404;
}
}
My problem is that the index.html file is served to the user but then the browser requests to access CSS and Javascript files are blocked because the browser cannot access the internal url.
Do you have any ideas to solve my problem?
I am also open to another way to serve a single-page application after backend authentication.
Thanks a lot.
You want to use the auth_request tag to make your life easier. Here is an example that you will need to retrofit unto your config. You could make your whole server require auth my just moving the auth_request tag to the top level outside of location
server {
...
location /docs {
auth_request /docs-auth;
...// Add your file redering here
}
location = /docs-auth {
internal;
proxy_pass http://auth-server;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-URI $request_uri;
}
}

X-Accel-Redirect Nginx configuration not working

I'm now deploying django projects on CentOS and have a problem with X-Accel-Redirect for protected file serving.
Here is my nginx.conf
server
{
listen 80;
server_name example;
index index.html index.htm index.php;
root /www/server/example;
charset UTF-8;
access_log /var/log/nginx/myproject_access.log;
error_log /var/log/nginx/myproject_error.log;
client_max_body_size 75M;
location /public/ {
root /www/wwwroot/myproject/;
}
location /media/ {
root /www/wwwroot/myproject/;
internal;
}
location / {
include uwsgi_params;
uwsgi_pass django;
}
...
}
Of course, protected files are on /www/wwwroot/myproject/media.
And corresponding python view file is following.
class ProtectedView(generics.GenericAPIView):
permission_classes = [IsAuthenticated]
def get(self, request, id, target):
file_name = "1.png"
response = HttpResponse()
response["X-Accel-Redirect"] = "/media/{0}".format(file_name)
response["Content-Disposition"] = "attachment; filename={0}".format(file_name)
return response
But server returns 404 error.
And the myproject_error.log is like this.
[error] 24570#0: *5 open() "/www/server/example/media/1.png" failed (2: No such file or directory), client: 174.11.13.81, server: example, request: "GET /protected-view/ HTTP/1.1", upstream: "uwsgi://0.0.0.0:8008", host: "40.1.12.23"
Maybe location /media/ {} block is not working. What problem? I have changed the permission but not working.
PS: I'm using django rest framework.

main view keeps redirecting to login page even for logged-in user in django

my main view is,
def main(request):
if request.user.is_authenticated:
return render(request,'main/main.html',{'use':use,'query':query,'noter':noter,'theme':request.user.profile.theme})
else:
return redirect('home')
the home page is responsible for serving login form is
def home(request):
if request.user.is_authenticated:
return redirect('main')
else:
return render(request,'home.html')
the "home.html" uses ajax to submit login page
$(document).ready(function(){
$('#form2').submit(function(event){
event.preventDefault();
$('#username_error').empty();
$("#password_error").empty();
var csrftoken = $("[name=csrfmiddlewaretoken]").val();
var formdata={
'username':$('input[name=username2]').val(),
'password':$('input[name=loginpassword]').val(),
};
$.ajax({
type:'POST',
url:'/Submit/logging',
data:formdata,
dataType:'json',
encode:true,
headers:{
"X-CSRFToken": csrftoken
},
})
.done(function(data){
if(!data.success){//we will handle error
if (data.password){
$('#password_error').text(data.password);
}
if(data.Message){
$('#password_error').text("You can't login via pc");
}
blocker();
return false;
}
else{
window.location='/';
}
});
event.preventDefault();
});
this configuration is working fine but main page keeps redirecting to home.html even when though user is already logged in. I'm using nginx server whose configuration is
server {
server_name host.me www.host.me;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/username/projectname;
}
location /media/ {
root /home/username/projectname/appname;
}
location / {
include proxy_params;
proxy_pass http://unix:/home/username/projectname/projectname.sock;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/www.host.me/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/www.host.me/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}server {
if ($host = www.host.me) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = host.me) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name host.me www.host.me;
return 404; # managed by Certbot
}
Edit 1
The urls of corresponding views are,
path('home',views.home,name='home'),
path('',views.main,name='main'),
path('Submit/logging',views.Loging,name='Loging'),
while my view for performing login is
def Loging(request):
if request.user_agent.is_pc:
return JsonResponse({'Message':'pc'})
else:
username1=request.POST['username']
print(username1)
username=username1.lower()
password=request.POST['password']
print(password)
user=authenticate(request,username=username,password=password)
if user is not None:
login(request,user)
return JsonResponse({'success':True})
else:
return JsonResponse({'password':'The user credentials do not exist in our database'})
Also if a user tries to access website from another url endpoint(say mywebsite.me/another) & already authenticated , then it will lead to the correct view which means login is performed correctly.

Django Rest Framework with nginx caching requests

Since I started using DRF with nginx/gunicorn server I've noticed that queries are being cached somehow. I.e. I got 10 records of object A in database (Postrgres) - if I ask for them with GET, I got 10 - ok. But then, when I add another record, I got 10 instead of 11 (although I see 10 in database). I have to manually reset gunicorn to get the correct result, which is not how REST should work...
Here's my nginx config (tried some cache settings):
upstream myapp{
server unix:/var/django/gunicorn.socket fail_timeout=0;
}
server {
listen 80;
server_name myapp.example.com;
client_max_body_size 4G;
access_log /var/django/myapp/logs/nginx-access.log;
error_log /var/django/myapp/logs/nginx-error.log;
# DRF API below
location /api {
expires off;
add_header Cache-Control "private, no-cache, must-revalidate, max-age=0";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_connect_timeout 75s;
proxy_read_timeout 300s;
proxy_pass http://myapp;
}
# JS for webapp (backbone)
location / {
alias /var/django/myapp_web/;
}
}
and gunicorn as well:
#!/bin/bash
NAME="myapp"
DJANGODIR=/var/django/myapp
SOCKFILE=/var/django/gunicorn.socket
USER=appuser
GROUP=appuser
NUM_WORKERS=5
DJANGO_SETTINGS_MODULE=myapp.settings
DJANGO_WSGI_MODULE=myapp.wsgi
TIMEOUT=300
echo "Starting $NAME as `whoami`"
cd $DJANGODIR
source /var/django/myapp/bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR
exec /var/django/myapp_venv/bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $NUM_WORKERS \
--reload \
--timeout $TIMEOUT \
--user=$USER --group=$GROUP \
--bind=unix:$SOCKFILE \
--log-level=debug \
--log-file=/var/django/myapp/logs/gunicorn-error.log
--access-logfile=/var/django/myapp/logs/gunicorn-access.log
EDIT
The viewset looks like nothing special:
class OrderViewSet(DefaultsMixin, viewsets.ModelViewSet):
serializer_class = OrderSerializer
queryset = Order.objects.all()
def get_queryset(self):
if self.request.GET:
filter_func = self.request.GET.get('filter', None)
allowed_filters = ['new', 'open', 'today', 'tomorrow', 'urgent', 'draft']
if filter_func and filter_func in allowed_filters:
if hasattr(self.queryset, filter_func):
self.queryset = getattr(self.queryset, filter_func)()
elif filter_func == 'draft':
self.queryset = Order.drafts.all()
if self.request.user.is_superuser:
return self.queryset
elif hasattr(self.request.user, 'driver'):
if self.request.user.is_staff:
return self.queryset
return self.queryset.for_driver(self.request.user.driver)
elif hasattr(self.request.user, 'employee'):
return self.queryset.for_employee(self.request.user.employee)
return self.queryset.filter(user = self.request.user)