Since I started using DRF with nginx/gunicorn server I've noticed that queries are being cached somehow. I.e. I got 10 records of object A in database (Postrgres) - if I ask for them with GET, I got 10 - ok. But then, when I add another record, I got 10 instead of 11 (although I see 10 in database). I have to manually reset gunicorn to get the correct result, which is not how REST should work...
Here's my nginx config (tried some cache settings):
upstream myapp{
server unix:/var/django/gunicorn.socket fail_timeout=0;
}
server {
listen 80;
server_name myapp.example.com;
client_max_body_size 4G;
access_log /var/django/myapp/logs/nginx-access.log;
error_log /var/django/myapp/logs/nginx-error.log;
# DRF API below
location /api {
expires off;
add_header Cache-Control "private, no-cache, must-revalidate, max-age=0";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_connect_timeout 75s;
proxy_read_timeout 300s;
proxy_pass http://myapp;
}
# JS for webapp (backbone)
location / {
alias /var/django/myapp_web/;
}
}
and gunicorn as well:
#!/bin/bash
NAME="myapp"
DJANGODIR=/var/django/myapp
SOCKFILE=/var/django/gunicorn.socket
USER=appuser
GROUP=appuser
NUM_WORKERS=5
DJANGO_SETTINGS_MODULE=myapp.settings
DJANGO_WSGI_MODULE=myapp.wsgi
TIMEOUT=300
echo "Starting $NAME as `whoami`"
cd $DJANGODIR
source /var/django/myapp/bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR
exec /var/django/myapp_venv/bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $NUM_WORKERS \
--reload \
--timeout $TIMEOUT \
--user=$USER --group=$GROUP \
--bind=unix:$SOCKFILE \
--log-level=debug \
--log-file=/var/django/myapp/logs/gunicorn-error.log
--access-logfile=/var/django/myapp/logs/gunicorn-access.log
EDIT
The viewset looks like nothing special:
class OrderViewSet(DefaultsMixin, viewsets.ModelViewSet):
serializer_class = OrderSerializer
queryset = Order.objects.all()
def get_queryset(self):
if self.request.GET:
filter_func = self.request.GET.get('filter', None)
allowed_filters = ['new', 'open', 'today', 'tomorrow', 'urgent', 'draft']
if filter_func and filter_func in allowed_filters:
if hasattr(self.queryset, filter_func):
self.queryset = getattr(self.queryset, filter_func)()
elif filter_func == 'draft':
self.queryset = Order.drafts.all()
if self.request.user.is_superuser:
return self.queryset
elif hasattr(self.request.user, 'driver'):
if self.request.user.is_staff:
return self.queryset
return self.queryset.for_driver(self.request.user.driver)
elif hasattr(self.request.user, 'employee'):
return self.queryset.for_employee(self.request.user.employee)
return self.queryset.filter(user = self.request.user)
Related
I'm serving django with gunicorn and nginx. And I've observed a strange behavior, which looks like django is prefixing MEDIA_URL with SCRIPT_NAME but it is not prefixing the STATIC_URL. Am I handling it properly?
My static's configuration:
STATIC_URL = "backend/static/" # this looks odd but works
STATIC_ROOT = "/var/www/static"
STATICFILES_DIRS = [BASE_DIR / "static"]
MEDIA_URL = "media/" # this looks fine
MEDIA_ROOT = "/var/www/media"
gunicorn configuration:
[Service]
User=www-data
Group=www-data
Environment="DJANGO_SETTINGS_MODULE=backend.settings.production"
WorkingDirectory=/var/www/backend
ExecStart=/var/www/backend/venv/bin/gunicorn \
--access-logfile - \
--workers 3 \
--bind unix:/run/gunicorn.sock \
backend.wsgi:application
nginx configuration:
location ^~ /backend {
proxy_set_header SCRIPT_NAME /backend;
include proxy_params;
proxy_pass http://unix:/run/gunicorn.sock;
location ^~ /backend/static {
alias /var/www/static/;
}
location ^~ /backend/media {
alias /var/www/media/;
}
}
//edit
from django sources
def __getattr__(self, name):
"""Return the value of a setting and cache it in self.__dict__."""
if (_wrapped := self._wrapped) is empty:
self._setup(name)
_wrapped = self._wrapped
val = getattr(_wrapped, name)
# Special case some settings which require further modification.
# This is done here for performance reasons so the modified value is cached.
if name in {"MEDIA_URL", "STATIC_URL"} and val is not None:
val = self._add_script_prefix(val)
elif name == "SECRET_KEY" and not val:
raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
self.__dict__[name] = val
return val
So why is it working so oddly?
I want to display my documentation (a single-page application built with React) only after authentication with my backend.
My configuration :
Nginx acts as a reverse proxy for the backend (Django) and serves static files like single-page-applications.
Django, the backend, identifies the user and makes a request to Nginx using X-Accel-Redirect.
So I proceed as follows:
1) Authentication on Django
views.py
def get_doc(request):
if request.method == 'POST':
form = PasswordForm(request.POST)
if form.is_valid():
if form.cleaned_data['password'] == 'foo':
response = HttpResponse()
response['Content-Type'] = ''
response['X-Accel-Redirect'] = '/docs-auth/'
return response
else:
return HttpResponse("Wrong password")
else:
form = PasswordForm()
return render(request, 'docs/form.html', {'form': form})
urls.py
urlpatterns = [
path('docs/', views.get_doc, name='documentation'),
]
2) Nginx serves the single-page application
upstream backend {
server web:8000;
}
server {
location = /favicon.ico {access_log off;log_not_found off;}
...
location /docs {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
proxy_pass http://backend;
}
location /docs-auth/ {
internal;
alias /home/foo/docs/;
index index.html;
try_files $uri $uri/ /docs/index.html;
}
location / {
alias /home/foo/landing_page/;
error_page 404 /404.html;
index index.html;
try_files $uri $uri/ =404;
}
}
My problem is that the index.html file is served to the user but then the browser requests to access CSS and Javascript files are blocked because the browser cannot access the internal url.
Do you have any ideas to solve my problem?
I am also open to another way to serve a single-page application after backend authentication.
Thanks a lot.
You want to use the auth_request tag to make your life easier. Here is an example that you will need to retrofit unto your config. You could make your whole server require auth my just moving the auth_request tag to the top level outside of location
server {
...
location /docs {
auth_request /docs-auth;
...// Add your file redering here
}
location = /docs-auth {
internal;
proxy_pass http://auth-server;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-URI $request_uri;
}
}
my main view is,
def main(request):
if request.user.is_authenticated:
return render(request,'main/main.html',{'use':use,'query':query,'noter':noter,'theme':request.user.profile.theme})
else:
return redirect('home')
the home page is responsible for serving login form is
def home(request):
if request.user.is_authenticated:
return redirect('main')
else:
return render(request,'home.html')
the "home.html" uses ajax to submit login page
$(document).ready(function(){
$('#form2').submit(function(event){
event.preventDefault();
$('#username_error').empty();
$("#password_error").empty();
var csrftoken = $("[name=csrfmiddlewaretoken]").val();
var formdata={
'username':$('input[name=username2]').val(),
'password':$('input[name=loginpassword]').val(),
};
$.ajax({
type:'POST',
url:'/Submit/logging',
data:formdata,
dataType:'json',
encode:true,
headers:{
"X-CSRFToken": csrftoken
},
})
.done(function(data){
if(!data.success){//we will handle error
if (data.password){
$('#password_error').text(data.password);
}
if(data.Message){
$('#password_error').text("You can't login via pc");
}
blocker();
return false;
}
else{
window.location='/';
}
});
event.preventDefault();
});
this configuration is working fine but main page keeps redirecting to home.html even when though user is already logged in. I'm using nginx server whose configuration is
server {
server_name host.me www.host.me;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/username/projectname;
}
location /media/ {
root /home/username/projectname/appname;
}
location / {
include proxy_params;
proxy_pass http://unix:/home/username/projectname/projectname.sock;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/www.host.me/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/www.host.me/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}server {
if ($host = www.host.me) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = host.me) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name host.me www.host.me;
return 404; # managed by Certbot
}
Edit 1
The urls of corresponding views are,
path('home',views.home,name='home'),
path('',views.main,name='main'),
path('Submit/logging',views.Loging,name='Loging'),
while my view for performing login is
def Loging(request):
if request.user_agent.is_pc:
return JsonResponse({'Message':'pc'})
else:
username1=request.POST['username']
print(username1)
username=username1.lower()
password=request.POST['password']
print(password)
user=authenticate(request,username=username,password=password)
if user is not None:
login(request,user)
return JsonResponse({'success':True})
else:
return JsonResponse({'password':'The user credentials do not exist in our database'})
Also if a user tries to access website from another url endpoint(say mywebsite.me/another) & already authenticated , then it will lead to the correct view which means login is performed correctly.
I am building a web API using Django and Nginx that needs to support Byte-Range requests.
Currently, when making a request such as:
curl --header "Content-Type: application/json"
--header "Authorization: Token 8f25cd3bb5b43a5277f4237b1d1db0273dbe8f6a"
--request POST http://myurl/download/
--header "Range: bytes=0-50"
-v
my Django view (the MyFileAPI class) is called, the user is authenticated, and the correct file path (/code/web/myapp/user_id/user_info.pbf) is fetched. However, the entire file is returned instead of only the first 50 bytes.
How do I configure Nginx and Django to allow partial file downloads?
default.conf
server {
listen 80;
server_name localhost;
charset utf-8;
client_max_body_size 10M;
add_header Accept-Ranges bytes;
proxy_force_ranges on;
location /static/ {
alias /django_static/;
}
location / {
include uwsgi_params;
uwsgi_pass web:8000;
}
}
nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
views.py
class MyFileAPI(APIView):
permission_classes = (IsAuthenticated,)
def post(self, request):
try:
user = request.user
my_file = MyFile.objects.filter(user_id=user).latest("created_date")
# ex) my_file.filepath = /myapp/13/user_info.pbf
# ex) my_files_path = /code/web/myapp/13/user_info.pbf
my_files_path = settings.BASE_DIR + my_file.filepath
response = FileResponse(open(my_files_path, 'rb'))
response['Accept-Ranges'] = 'bytes'
response['X-Accel-Buffering'] = 'no'
# Adding this next line results in a 404 error
# response['X-Accel-Redirect'] = my_files_path
return response
except Exception as e:
return Response({'status': str(e)}, content_type="application/json")
When I add the line response['X-Accel-Redirect'] = my_files_path I receive the error The current path /myapp/13/user_info.pbf didn't match any of these.
and all of my urls from urls.py are listed.
I know there are other posts on the same topic, but none give a full answer.
The following is my python code
from gevent import monkey;monkey.patch_all()
from flask import Flask,render_template, url_for, request, redirect, flash,jsonify,session,Markup
from socketio import socketio_manage
from socketio.namespace import BaseNamespace
from socketio.server import SocketIOServer
app=Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'
class ChatNamespace(BaseNamespace):
def recv_connect(self):
print "successfully connected"
self.emit('show_result','successfully connect')
def on_receive_message(self,msg):
print "message is "+msg["data"]
self.emit('show_result2',msg["data"])
#app.route('/')
def index():
#print "in the function"
return render_template('index.html')
#app.route("/socket.io/<path:path>")
def run_socketio(path):
socketio_manage(request.environ, {'/test': ChatNamespace})
return 'ok'
if __name__=='__main__':
#app.run(debug=True, port=80, host='0.0.0.0')
app.debug=True
#app.run()
SocketIOServer(('0.0.0.0', 5000), app,resource="socket.io").serve_forever()
print "successfull listen to socket"
and the following is nginx configuration
server {
location / {
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
try_files $uri #proxy;
}
location #proxy {
proxy_pass http://127.0.0.1:8000;
}
location /templates {
alias /home/www/flask_project/templates/;
}
location /script {
alias /home/www/flask_project/script/;
}
location /static {
alias /home/www/flask_project/static/;
}
}
Each time I run the app ,I use the following command
gunicorn main2:app -b localhost:5000
I know that I am missing a lot of information to run this gevent-socketio app on a live server.
Can anyone help me out, I am totally new to this web-socket technology
Have you tried FlaskSocketio??
If you are trying to do it without the extension, you will need the socketio gunicorn worker to run your app.
gunicorn --worker-class socketio.sgunicorn.GeventSocketIOWorker module:app