I'm serving django with gunicorn and nginx. And I've observed a strange behavior, which looks like django is prefixing MEDIA_URL with SCRIPT_NAME but it is not prefixing the STATIC_URL. Am I handling it properly?
My static's configuration:
STATIC_URL = "backend/static/" # this looks odd but works
STATIC_ROOT = "/var/www/static"
STATICFILES_DIRS = [BASE_DIR / "static"]
MEDIA_URL = "media/" # this looks fine
MEDIA_ROOT = "/var/www/media"
gunicorn configuration:
[Service]
User=www-data
Group=www-data
Environment="DJANGO_SETTINGS_MODULE=backend.settings.production"
WorkingDirectory=/var/www/backend
ExecStart=/var/www/backend/venv/bin/gunicorn \
--access-logfile - \
--workers 3 \
--bind unix:/run/gunicorn.sock \
backend.wsgi:application
nginx configuration:
location ^~ /backend {
proxy_set_header SCRIPT_NAME /backend;
include proxy_params;
proxy_pass http://unix:/run/gunicorn.sock;
location ^~ /backend/static {
alias /var/www/static/;
}
location ^~ /backend/media {
alias /var/www/media/;
}
}
//edit
from django sources
def __getattr__(self, name):
"""Return the value of a setting and cache it in self.__dict__."""
if (_wrapped := self._wrapped) is empty:
self._setup(name)
_wrapped = self._wrapped
val = getattr(_wrapped, name)
# Special case some settings which require further modification.
# This is done here for performance reasons so the modified value is cached.
if name in {"MEDIA_URL", "STATIC_URL"} and val is not None:
val = self._add_script_prefix(val)
elif name == "SECRET_KEY" and not val:
raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
self.__dict__[name] = val
return val
So why is it working so oddly?
Related
I have setup my python project using nginx and gunicorn as described in this tutorial https://djangocentral.com/deploy-django-with-nginx-gunicorn-postgresql-and-lets-encrypt-ssl-on-ubuntu/
so far everything works perfectly fine.
My project is in /root/project_folder/
I want to upload files via the admin page, but I get a bad request error 400. The media folder I want to add the files to is /var/www/my_domain/media (owned by the group www-data). This is also properly configured since I can see the images when I move them manually into that folder.
Do you guys maybe have any idea why the issue may be ?
the main problem is when uploading I need to save the image to the media root:
/var/www/domain/media/images/dad/ProfilePicture/offline_screen.png
But when I send request to view the image the web server returns the following:
/media/var/www/domain/media/images/dad/ProfilePicture/offline_screen.png
The upload_to path needs to be images/dad/ProfilePicture/offline_screen.png for correct request but then the image is uploaded to the wrong folder
Any ideas ? Thanks in advance!
UPDATE:
Here is my nginx config
`
server {
server_name domain.net www.domain.net ip_address;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /var/www/domain;
}
location = /media/ {
root /var/www/domain;
}
location / {
include proxy_params;
proxy_pass http://unix:/var/log/gunicorn/domain.sock;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/domain/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/domain/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = www.domain) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = domain) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name domain www.domain ip_address;
return 404; # managed by Certbot
}
`
Here is my media root / media url:
MEDIA_ROOT = '/var/www/domain/media' #os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'
The url.py:
urlpatterns = [
path('admin/', admin.site.urls),
path('', include('webpages.urls')),
]
if settings.DEBUG:
urlpatterns += static(settings.MEDIA_URL,document_root=settings.MEDIA_ROOT)
urlpatterns += static(settings.STATIC_URL,document_root=settings.STATIC_ROOT)
And finally how I serve and upload Images:
def get_path_of_content_image(instance, filename):
return os.path.join(settings.MEDIA_ROOT, "images", str(instance.course.topic), str(instance.course), filename)
class Topic(models.Model):
title = models.CharField(max_length=100)
description = models.TextField(max_length=445)
image = models.ImageField(upload_to=get_topic_profile_path, default="images/default.jpeg")
def __str__(self):
return self.title
def image_tag(self):
if self.image:
return mark_safe('<img src="%s" style="width: 200px; height:200px;" />' % self.image.get_image())
def get_image(self):
if self.image:
return str(self.image.url).replace('/media/var/www/domain', '') ########1
else:
return settings.STATIC_ROOT + 'webpages/img/default.jpeg'
image_tag.short_description = 'Current Image'
I wrote this line #######1 since set the upload path to the full media root but then in order to return the correct path I need to remove the media root prefix. And also very importantly this only works if DEBUg set to TRUE.
I'm not certain this will solve your issue, but there are 2 things that need to be configured correctly here. If nginx is serving the files in your media directory ok, I'm guessing your nginx config is already ok and you just need to edit settings.py, but I'll include the nginx config in case it helps someone else.
UPDATE
I don't think those methods in your models are necessary, although I don't really know how they are used. Here's an example of how I configure a model with an ImageField:
Firstly, how to upload and save images
(letting django do most of the work)
models.py
...
class Post(models.Model):
"""Represents a single blog post"""
# model fields
title = models.CharField(max_length=150, unique=True)
content = models.TextField()
title_image = models.ImageField(upload_to='image/', blank=True, null=True)
...
forms.py
...
class PostModelForm(forms.ModelForm):
class Meta:
model = Post
fields = ['title', 'title_image', 'content']
widgets = {
'content': Textarea(attrs={'cols': 80, 'rows': 25}),
}
...
views.py
...
def new_post(request):
context = {}
template = 'new_post.html'
# POST
if request.method == 'POST':
form = PostModelForm(request.POST or None, request.FILES)
if form.is_valid():
new_post = form.save()
return redirect('list_post')
else:
context['form'] = form
return render(request, template, context)
# GET
else:
context['form'] = PostModelForm()
return render(request, template, context)
...
new_post.html (note the enctype of the form - important if you want to upload files)
...
<div class="card bg-light">
<div class="card-header">
<h3>New Post</h3>
</div>
<div class="card-body">
<form action="{% url 'blog:new_post' %}" method="POST" class="w-100" enctype="multipart/form-data">
<div class="col-12">
{% csrf_token %}
{{ form.as_p }}
</div>
<div class="col-12">
<ul class="actions">
<button class="primary" type="submit">Save</button>
</ul>
</div>
</form>
</div>
</div>
...
Secondly, how to serve your images
Now you want to display the image in you markup
In views.py
...
def detail(request, id):
context = {}
template_name = 'post.html'
post = get_object_or_404(Post, id=id)
context['post'] = post
return render(request, template_name, context)
...
In your markup in post.html
...
{% if post.title_image %}
<span class="img"><img src="{{ post.title_image.url }}" alt="image for {{ post.title }}" /></span>
{% endif %}
...
(1) Define MEDIA_ROOT and MEDIA_URL in settings.py
In settings.py add
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'
Note: Read the django docs on MEDIA_ROOT and MEDIA_URL
(2) define media location in nginx server block
In that tutorial the nginx configuration doesn't define a location for media. Here's the server block from the tutorial:
server {
listen 80;
server_name server_domain_or_IP;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /path/to/staticfiles;
}
location / {
include proxy_params;
proxy_pass http://unix:/var/log/gunicorn/project_name.sock;
}
}
I think it should look something like this:
server {
listen 80;
server_name server_domain_or_IP;
location = /favicon.ico { access_log off; log_not_found off; }
location /media/ {
root /path/to/media;
}
location /static/ {
root /path/to/staticfiles;
}
location / {
include proxy_params;
proxy_pass http://unix:/var/log/gunicorn/project_name.sock;
}
}
If that configuration doesn't work, I've posted what works for me below
In my own experience with django, nginx and gunicorn however, my nginx configuration looks like this:
server {
listen 443 ssl;
listen [::]:443 ssl;
ssl_certificate /etc/letsencrypt/live/website_name.com-0001/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/website_name.com-0001/privkey.pem; # managed by Certbot
server_name website_name.com www.website_name.com;
location = /favicon.ico { access_log off; log_not_found off; }
location /media {
alias /home/user_name/website_name/project_name/media;
}
location /static/ {
alias /home/user_name/website_name/project_name/static_root/;
}
location / {
include proxy_params;
proxy_pass http://unix:/run/gunicorn.sock;
}
}
Note 1: Here are the nginx docs for alias and root directives.
So after spending an eternity on debugging, I have the solution to my problem:
First Thing that was problematic is the '=' between 'location' and '/media/' in the nginx file. Be sure that this is not root of the error. It should look like this:
location /media {
alias /home/user_name/website_name/project_name/media;
}
and not like this:
location = /media {
alias /home/user_name/website_name/project_name/media;
}
The file upload to a custom local directory can be done like this:
from django.core.files.storage import FileSystemStorage
fs = FileSystemStorage(location=settings.MEDIA_ROOT)
and for the model add the file storage option on upload (or change the name of the image with the updated path afterwards):
image = models.ImageField(upload_to=get_topic_profile_path, default="images/default.jpeg", storage=fs)
thanks for the other answers. I learned quite a lot by fixing this erros :D
I am trying to run a flask web app and a flask restful api on the same domain.
They both run fine, when they are run alone.
I am not using venv.
However, when I try to run them under a same domain, Nginx throws an error - 404, not found. Also, localhost/ runs the web-app fine without errors but localhost/web and localhost/rest gives a 404.
The request.url prints http://localhost/web and http://localhost/rest but request is not being sent to flask server responsible for `/rest', it seems, as no data is logged for the second server (the restful api). All logged data is for the 1st server (the web app) only.
#Nginx.conf:
server {
listen 80;
server_name localhost;
location /web {
include uwsgi_params;
uwsgi_pass unix:/var/www/webproject/web_uwsgi.sock;
}
location /api {
include uwsgi_params;
uwsgi_pass unix:/var/www/restproject/rest_api_uwsgi.sock;
}
}
#webproject.ini #restproject.ini
[uwsgi] [uwsgi]
vhost = true vhost = true
project = webproject project = restproject
base = /var/www/ base = /var/www/
chdir = %(base)%(project) chdir = %(base)%(project)
wsgi-file = run.py wsgi-file = api.py
callable = app callable = ap
mount = /webproject = app mount = /restproject = ap
manage-script-name = true manage-script-name = true
plugins = python plugins = python
socket = /var/www/webproject/%n. socksocket = /var/www/restproject/%n.sock
chmod-socket = 666 chmod-socket = 666
logto = /var/log/uwsgi/%n.log logto = /var/log/uwsgi/%n.log
Any suggestion would be helpful.
Since I started using DRF with nginx/gunicorn server I've noticed that queries are being cached somehow. I.e. I got 10 records of object A in database (Postrgres) - if I ask for them with GET, I got 10 - ok. But then, when I add another record, I got 10 instead of 11 (although I see 10 in database). I have to manually reset gunicorn to get the correct result, which is not how REST should work...
Here's my nginx config (tried some cache settings):
upstream myapp{
server unix:/var/django/gunicorn.socket fail_timeout=0;
}
server {
listen 80;
server_name myapp.example.com;
client_max_body_size 4G;
access_log /var/django/myapp/logs/nginx-access.log;
error_log /var/django/myapp/logs/nginx-error.log;
# DRF API below
location /api {
expires off;
add_header Cache-Control "private, no-cache, must-revalidate, max-age=0";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_connect_timeout 75s;
proxy_read_timeout 300s;
proxy_pass http://myapp;
}
# JS for webapp (backbone)
location / {
alias /var/django/myapp_web/;
}
}
and gunicorn as well:
#!/bin/bash
NAME="myapp"
DJANGODIR=/var/django/myapp
SOCKFILE=/var/django/gunicorn.socket
USER=appuser
GROUP=appuser
NUM_WORKERS=5
DJANGO_SETTINGS_MODULE=myapp.settings
DJANGO_WSGI_MODULE=myapp.wsgi
TIMEOUT=300
echo "Starting $NAME as `whoami`"
cd $DJANGODIR
source /var/django/myapp/bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR
exec /var/django/myapp_venv/bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $NUM_WORKERS \
--reload \
--timeout $TIMEOUT \
--user=$USER --group=$GROUP \
--bind=unix:$SOCKFILE \
--log-level=debug \
--log-file=/var/django/myapp/logs/gunicorn-error.log
--access-logfile=/var/django/myapp/logs/gunicorn-access.log
EDIT
The viewset looks like nothing special:
class OrderViewSet(DefaultsMixin, viewsets.ModelViewSet):
serializer_class = OrderSerializer
queryset = Order.objects.all()
def get_queryset(self):
if self.request.GET:
filter_func = self.request.GET.get('filter', None)
allowed_filters = ['new', 'open', 'today', 'tomorrow', 'urgent', 'draft']
if filter_func and filter_func in allowed_filters:
if hasattr(self.queryset, filter_func):
self.queryset = getattr(self.queryset, filter_func)()
elif filter_func == 'draft':
self.queryset = Order.drafts.all()
if self.request.user.is_superuser:
return self.queryset
elif hasattr(self.request.user, 'driver'):
if self.request.user.is_staff:
return self.queryset
return self.queryset.for_driver(self.request.user.driver)
elif hasattr(self.request.user, 'employee'):
return self.queryset.for_employee(self.request.user.employee)
return self.queryset.filter(user = self.request.user)
I'm trying to host my Django Static and Media files on Amazon S3 and I've been following every guide out there, but I still end up getting S3ResponseError: 301 Moved Permanently errors on deployment of my Elastic Beanstalk Application when it tries to run collectstatic.
My S3 is working and I can access other files on it. I also have it set to a custom domain so you can access the same file in the following ways:
http://s3.condopilot.com.s3-eu-west-1.amazonaws.com/thumbs/big/3fca62e2150e8abec3f693a6eae8d2f79bb227fb.jpg
https://s3-eu-west-1.amazonaws.com/s3.condopilot.com/thumbs/big/3fca62e2150e8abec3f693a6eae8d2f79bb227fb.jpg
http://s3.condopilot.com/thumbs/big/3fca62e2150e8abec3f693a6eae8d2f79bb227fb.jpg
It is the third option that I want to use, but I've tried the other ones aswell. Both with and without https:// in the settings below.
My settings file look like this
#settings.py file
AWS_ACCESS_KEY_ID = 'XXX'
AWS_SECRET_ACCESS_KEY = 'XXX'
AWS_HEADERS = {
'Expires': 'Thu, 31 Dec 2099 20:00:00 GMT',
'Cache-Control': 'max-age=94608000',
}
AWS_STORAGE_BUCKET_NAME = 's3.condopilot.com'
# I have also tried setting AWS_S3_CUSTOM_DOMAIN to the following:
# - "s3-eu-west-1.amazonaws.com/%s/" % AWS_STORAGE_BUCKET_NAME
# - "s3-eu-west-1.amazonaws.com/%s" % AWS_STORAGE_BUCKET_NAME
# - "s3.condopilot.com"
AWS_S3_CUSTOM_DOMAIN = "%s.s3-eu-west-1.amazonaws.com" % AWS_STORAGE_BUCKET_NAME
AWS_S3_CALLING_FORMAT = 'boto.s3.connection.OrdinaryCallingFormat'
AWS_S3_SECURE_URLS = False # Tried both True and False
AWS_S3_URL_PROTOCOL = 'http' # Tried with and without
STATICFILES_LOCATION = 'static'
STATIC_URL = "http://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
MEDIAFILES_LOCATION = 'media'
MEDIA_URL = "http://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
The reason I have AWS_S3_CALLING_FORMAT = 'boto.s3.connection.OrdinaryCallingFormat' is because without it I get the following error:
ssl.CertificateError: hostname 's3.condopilot.com.s3.amazonaws.com' doesn't match either of '*.s3.amazonaws.com', 's3.amazonaws.com'. All advice I find online regarding that error says that OrdinaryCallingFormat should be used when bucket name contains dots, example s3.condopilot.com.
My custom storages looks like this
#custom_storages.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
And yes, my S3 bucket is set up in eu-west-1.
I think you do not need to set the region S3 in the URL and if you are using django-storage replaces this app to django-storages-redux. You don't need the custom_storages.py file.
Keep things simple. This is enough.
from django.utils import six
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXX'
AWS_SECRET_ACCESS_KEY = 'XxXxXxXxXxXxXxXxXxXxXxXxXxXxxXxX'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
AWS_AUTO_CREATE_BUCKET = False
AWS_QUERYSTRING_AUTH = False
AWS_EXPIRY = 60 * 60 * 24 * 7
AWS_HEADERS = {
'Cache-Control': six.b('max-age=%d, s-maxage=%d, must-revalidate' % (
AWS_EXPIRY, AWS_EXPIRY))
}
MEDIA_URL = 'https://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
STATICFILES_STORAGE = DEFAULT_FILE_STORAGE
STATIC_URL = MEDIA_URL
The following is my python code
from gevent import monkey;monkey.patch_all()
from flask import Flask,render_template, url_for, request, redirect, flash,jsonify,session,Markup
from socketio import socketio_manage
from socketio.namespace import BaseNamespace
from socketio.server import SocketIOServer
app=Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'
class ChatNamespace(BaseNamespace):
def recv_connect(self):
print "successfully connected"
self.emit('show_result','successfully connect')
def on_receive_message(self,msg):
print "message is "+msg["data"]
self.emit('show_result2',msg["data"])
#app.route('/')
def index():
#print "in the function"
return render_template('index.html')
#app.route("/socket.io/<path:path>")
def run_socketio(path):
socketio_manage(request.environ, {'/test': ChatNamespace})
return 'ok'
if __name__=='__main__':
#app.run(debug=True, port=80, host='0.0.0.0')
app.debug=True
#app.run()
SocketIOServer(('0.0.0.0', 5000), app,resource="socket.io").serve_forever()
print "successfull listen to socket"
and the following is nginx configuration
server {
location / {
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
try_files $uri #proxy;
}
location #proxy {
proxy_pass http://127.0.0.1:8000;
}
location /templates {
alias /home/www/flask_project/templates/;
}
location /script {
alias /home/www/flask_project/script/;
}
location /static {
alias /home/www/flask_project/static/;
}
}
Each time I run the app ,I use the following command
gunicorn main2:app -b localhost:5000
I know that I am missing a lot of information to run this gevent-socketio app on a live server.
Can anyone help me out, I am totally new to this web-socket technology
Have you tried FlaskSocketio??
If you are trying to do it without the extension, you will need the socketio gunicorn worker to run your app.
gunicorn --worker-class socketio.sgunicorn.GeventSocketIOWorker module:app