Read static file in view - django

To integrate Django and Ember, I have decided to serve my Ember SPA in a Django view (avoids CORS issues, only one server for frontend and API, etc). I do it like this:
# urls.py
urlpatterns = [
url(r'^admin/', include(admin.site.urls)),
url(r'^api/', include(api_urls, namespace='api')),
...
url(r'^$', views.emberapp, name='emberapp'),
...
]
# views.py
from django.http import HttpResponse
def emberapp(request):
# The Ember frontend SPA index file
# This only works in development, and is anyway hacky
EMBER_FE_INDEX_HTML = '/absolute/path/to/my/frontend/static/fe-dist/index.html'
template_file = open(EMBER_FE_INDEX_HTML)
html_content = index_file.read()
index_file.close()
return HttpResponse(html_content)
The index.html is part of the static assets. In development this is very easy:
The index.html is directly accessible to the Django application in the file system
I know the absolute path to the index file
But in production things are more complex, because the static assets are not local to the django application, but accessible on Amazon S3. I use django-storages for that.
How can I read the contents of a static file from a view, in a generic way, no matter what backend is used to store/serve the static files?

First, I don't think the way you do it is a good idea.
But, to answer your question: In your settings.py, you likely have defined the directory where Django will collect all static files.
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
So in your view, you just need to fetch the file os.path.join(settings.STATIC_ROOT, 'index.html')
That said, you should serve index.html via the webserver, same as your static/ files, robots.txt, favicon.ico, etc. Not through Django. The webserver is much faster, uses proper caching, and its just one line in your Nginx or Apache settings, instead of an entire view function in Django.

This is my current solution. Works in development, not sure about production yet (it is a pain that you need to commit untested code to verify production-related code in Heroku)
from django.conf import settings
from django.http import HttpResponse
from django.core.files.storage import get_storage_class
FE_INDEX_HTML = 'fe/index.html' # relative to the collectstatic directory
def emberapp(request):
# The Ember frontend SPA index file
# By getting the storage_class like this, we guarantee that this will work
# no matter what backend is used for serving static files
# Which means, this will work both in development and production
# Make sure to run collectstatic (even in development)
# TODO: how to use this in development without being forced to run collectstatic?
storage_class = get_storage_class(settings.STATICFILES_STORAGE)
# TODO: reading from a storage backend can be slow if assets are in a third-party server (like Amazon S3)
# Maybe streaming the static file from the server would be faster?
# No redirect to the Amazon S3 asset, please, since the Ember App needs to
# run from the same URL as the API, otherwise you get CORS issues
with storage_class().open(FE_VWORKS_INDEX_HTML) as index_file:
html_content = index_file.read()
return HttpResponse(html_content)
Or, to reply with an StreamingHttpResponse, which does not force Django to read the whole file in memory (and wait for it to be read):
def emberapp(request):
# The Ember frontend SPA index file
# By getting the storage_class like this, we guarantee that this will work
# no matter what backend is used for serving static files
# Which means, this will work both in development and production
# Make sure to run collectstatic (even in development)
# TODO: how to use this in development without being forced to run collectstatic?
storage_class = get_storage_class(settings.STATICFILES_STORAGE)
index_file = storage_class().open(FE_INDEX_HTML)
return StreamingHttpResponse(index_file)

Related

Why doesn't Django serve static files when the dubug=False in settings?

when I tried to run the project, static files are missing, and the settings.debug is set to False.
Django does not serve static files in production (when DEBUG = False) because doing so would be in their own words (As stated in the section Serving static files during development of the documentation):
This method is grossly inefficient and probably insecure, so
it is unsuitable for production.
Why is this inefficient one might say? Well static files are mostly larger in size than your normal HTML files, plus a website is bound to have lots of static files. If Django served static content even in production much of the time of the server would be wasted in serving these static files. Plus to serve multiple requests at the same time we run multiple Django processes simultaneously, if there are many requests for static files this will cause the processes to waste time serving them, causing other requests to wait if there are no free processes.
Plus as #Reda Bourial mentions in their comment Django doesn't handle compression well (One would want to compress their static files so that less bandwidth is required, both by the server and the client). Furthermore Django should focus more on the task it is designed for, which is rendering the pages requested by users (A CPU bound task), whereas serving static files is mostly just an Input / Output (I/O) task and for Django to spend time on these tasks (even when it is not efficient at them) is clearly a waste.
Servers like NGINX or Apache can serve static files much more efficiently, hence it is best to use them in production instead of having Django do it. For details on how to configure static files for production see Deploying static files [Django docs]
Actually in Django, while you run your project on the development server it won't serve your static files when you try to serve the static files after changing DEBUG settings to False. It will work properly on the production or stagging server but not on localhost server.
Still, there is a way to do this you can get by running your development server on insecure mode.
Try this,
python manage.py runserver --insecure
it is not recommended, but if you are just getting started and want to understand more about what's going on, then you can also link directly:
urls.py:
from django.conf.urls.static import static
from django.contrib import admin
from djangoapp import settings
from webtools.views import *
from django.urls import path, include
"""
direct link on the server in production
"""
from django.views.static import serve as mediaserve
from django.conf.urls import url
urlpatterns = [
path('admin/', admin.site.urls),
path('', include('webtools.urls')),
]
if settings.DEBUG:
import debug_toolbar
urlpatterns = [
path('__debug__/', include(debug_toolbar.urls)),
] + urlpatterns
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
else:
# direct link on server in production
urlpatterns += [
url(f'^{settings.MEDIA_URL.lstrip("/")}(?P<path>.*)$', mediaserve, {'document_root': settings.MEDIA_ROOT}),
url(f'^{settings.STATIC_URL.lstrip("/")}(?P<path>.*)$', mediaserve, {'document_root': settings.STATIC_ROOT}),
]
settings.py:
DEBUG = False
.
.
.
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = []
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

How To Serve media Files In Production

I have a Django project which sends automated e-mails with attached pdfs to users. At the moment I just have a normal /media/ folder with the pdf in it which the code points to. This works in development but throws a server error in production.
My question is how do I server media files in production? I have read a lot about it but can't find exactly what I'm after.
I use collectstatic in my production environment which works for static files, I'd expect something similar for media files.
urls
urlpatterns = [
path('admin/', admin.site.urls),
path('', include('page.urls')),
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
settings
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static_files')
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, "static_media")
views.py (that sends the file)
file_path = os.path.join(settings.STATIC_ROOT+'\\pdf\\free_pdf.pdf')
...
msg.attach_file(file_path)
passenger.wsgi
import os
import sys
sys.path.append(os.getcwd())
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'
import django.core.handlers.wsgi
from django.core.wsgi import get_wsgi_application
from whitenoise import WhiteNoise
SCRIPT_NAME = os.getcwd()
SCRIPT_NAME = '' #solution for damn link problem
class PassengerPathInfoFix(object):
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
from urllib.parse import unquote
environ['SCRIPT_NAME'] = SCRIPT_NAME
request_uri = unquote(environ['REQUEST_URI'])
script_name = unquote(environ.get('SCRIPT_NAME', ''))
offset = request_uri.startswith(script_name) and len(environ['SCRIPT_NAME']) or 0
environ['PATH_INFO'] = request_uri[offset:].split('?', 1)[0]
return self.app(environ, start_response)
application = get_wsgi_application()
application = PassengerPathInfoFix(application)
application = WhiteNoise(application, root='/home/mysite/mysite/static_files')
(this code is from when I abandoned the media folder and was trying to just serve it with my static files)
My proj structure:
|project
|__app
|__static
| |__app
| |__style.css
|__media
| |__app
| |__pdfToBeSent
|__static_files (generated by collectstatic)
|__media_files (i created this)
collectstatic also doesn't copy my projects media files into media_files, I have been copying them in there myself (unsure about this).
I have also tried moving the media file to every possible location in my project
I am also serving my static files with whitenoise.
Thank you.
Since I was in a similar situation (on the same hosting, A2Hosting), I will try to share my understanding of the problem and the solution I opted for.
Pardon me if I may seem presumptuous in this presentation, I'm simply trying to retrace all the points that represent the flow of thoughts that led me to this solution.
A small premise: if with "media files" you intend multimedial files, such as images and so on, I think you shouldn't use the Django media folder as it's designed to serve files uploaded by the users.
Not knowing if your PDFs are indeed uploaded by some user or not, I'll try to expose my solution anyway.
When in a production environment, Django isn't going to serve static and media files.
For the former I too used WhiteNoise, while for the latter the approach is different (at least on a shared hosting base).
When we set Debug = False in settings.py I suspect that the media folder, created along with the Django project, becomes somewhat unreadable/unaccessible (I cannot tell if by the hand of Django or by the hand of the hosting, or a conjuction of the two).
The official method to handle media files is indeed to rely on an external storage service (like Amazon S3), but this solution isn't suitable for budget limited scenarios.
In my situation, I had the Django app running on a subdomain related to my main domain.
Principal domain: www.mydomain.com
App domain: subdomain.mydomain.com
Leaving the media folder created with the Django project where it was, I created another one in the following unix path:
/home/A2hosting_username/subdomain.mydomain.com/media
Then, in settings.py I changed the MEDIA_ROOT variable from:
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
to:
MEDIA_ROOT = '/home/A2hosting_username/subdomain.mydomain.com/media'
After that, in the model.py class used to define the database and interact with it, I specified the path to the uploaded media (videos, in my case) in this way:
class Video(models.Model):
name = models.CharField(max_length=50)
path = models.FileField(upload_to="../media/videos", null=True, verbose_name="")
Since we don't have access to the Apache config file, it may be useful to edit the .htaccess file (relative to subdomain.mydomain.com) in the following way, to prevent the browser to "time out" when the uploaded file is somewhat heavy:
<ifModule mod_headers.c>
Header set Connection keep-alive
</ifModule>
I hope this can be of any help.

How to fix cors error when loading static file served by Django runserver

Relevant info : Django version 2.2
I have installed django-cors-headers and added it to enabled apps and added the middleware to middleware settings as first middleware.
I have also configured the settings:
INSTALLED_APPS = [
...
"corsheaders",
"rest_framework",
"django.contrib.admin",
...
]
MIDDLEWARE = [
...
"corsheaders.middleware.CorsMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
"django.middleware.common.CommonMiddleware",
...
]
CORS_ORIGIN_ALLOW_ALL = True
CORS_ALLOW_CREDENTIALS = True
My setup has 2 projects running on different ports. Django server running on 8000 and my front-end running on 8080.
All API requests the front-end sends against back-end work just fine. But when I try to load one.js file served by the back-end project it fails with this error:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://127.0.0.1:8000/static/myscript.js. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing)
The way I load the script is by creating a new script element and add the URL http://127.0.0.1:8000/static/myscript.js as value for the src attribute on the script tag.
This error does not occur when I use a value like https://code.jquery.com/jquery-3.5.1.min.js instead of http://127.0.0.1:8000/static/myscript.js so it looks like there is some issue with static files being served by Django runserver, but what exactly and why is beyond me.
I know that there is a bunch of cors related issues like Django CORS on static asset, but they are not about my issue or have gone unanswered. Also, I noticed, that the requests going at static files bypass the cors middleware anyway, so perhaps I should be doing something else here instead?
Perhaps someone can help me out here?
Thanks
A slightly different approach based on Odif Yitsaeb's idea, however you don't need to remove staticfiles or mess with urlpatterns. Simply place the following code into your settings.py:
from django.contrib.staticfiles import handlers
# extend StaticFilesHandler to add "Access-Control-Allow-Origin" to every response
class CORSStaticFilesHandler(handlers.StaticFilesHandler):
def serve(self, request):
response = super().serve(request)
response['Access-Control-Allow-Origin'] = '*'
return response
# monkeypatch handlers to use our class instead of the original StaticFilesHandler
handlers.StaticFilesHandler = CORSStaticFilesHandler
Notes:
Don't use that in production (you shouldn't use devserver in production anyway)
The handler must be monkeypatched very early, before any requests are served. Placing the code in settings.py will do the trick
It looks like I have sort of found the answer:
What I did was I added INSTALLED_APPS.remove('django.contrib.staticfiles') into dev settings.
and I created view like
from django.contrib.staticfiles.views import serve
def cors_serve(request, path, insecure=False, **kwargs):
kwargs.pop('document_root')
response = serve(request, path, insecure=insecure, **kwargs)
response['Access-Control-Allow-Origin'] = '*'
return response
for serving static content. And I added it into urlconf like this:
from utils.views import cors_serve
...
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT, view=cors_serve)
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT, view=cors_serve)
This does take care of the issue for me. I know that the Django docs (https://docs.djangoproject.com/en/2.2/ref/contrib/staticfiles/#runserver) suggest that you can skip static files serving with using runserver like this: django-admin runserver --nostatic but that had absolutely no effect for me if I still had "django.contrib.staticfiles" in my INSTALLED_APPS

Adding expire header for Django static files on Heroku

I'm trying to optimize my webpage and am having trouble setting expiration date headers on my static files.
I am running django-1.5, python-2.7.3.
Here's my cache settings in settings.pyso far:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
'LOCATION': os.path.join(PROJECT_ROOT, 'cache/'),
}
}
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_SECONDS = 5 * 60
CACHE_MIDDLEWARE_KEY_PREFIX = ''
MIDDLEWARE_CLASSES = (
'django.middleware.cache.UpdateCacheMiddleware',
...
'django.middleware.cache.FetchFromCacheMiddleware',
)
And my static file settings in settings.py:
import os.path
PROJECT_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_ROOT = os.path.abspath(os.path.join(PROJECT_DIR, '..'))
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'staticfiles/')
STATIC_URL = '/static/'
STATICFILES_DIRS = (
os.path.join(PROJECT_DIR, 'static'),
)
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
# 'django.contrib.staticfiles.finders.DefaultStorageFinder',
)
The closest advice I've found was here, but I'm unable to modify the .htaccess files on Heroku.
Any help is greatly appreciated. Thanks!
The django staticfiles app does not provide out of the box support for custom headers. You'll have to hack together your own view to serve up the files and add custom headers to the HttpResponse.
But You should not be serving your static files using Django. This is a terrible idea.
Django is single-threaded, and blocking. So every time you're serving a user a static file, you're literally serving nothing else (including your application code, which is what Django is there for).
Django's staticviews file is insecure, and unstable. The documentation specifically says not to use it in production. So do not use it in production. Ever.
In production you shouldn't serve static files with Django. See the warning boxes on this page: https://docs.djangoproject.com/en/1.4/ref/contrib/staticfiles/#static-file-development-view
In development, Django's contrib.staticfiles app automatically serves staticfiles for you by overwriting the runserver command. This way you can't control the way it serves the static files.
You can prevent the staticfiles app from serving the static files by adding the --nostatic option to the runserver command:
./manage.py runserver --nostatic
Then you can write an url config to manually serve the static files with headers you want:
from functools import wraps
from django.conf import settings
from django.contrib.staticfiles.views import serve as serve_static
from django.conf.urls import patterns, url
urlpatterns = patterns('', )
if settings.DEBUG:
def custom_headers(view_func):
#wraps(view_func)
def wrapper(request, *args, **kwargs):
response = view_func(request, *args, **kwargs)
response['Custom-header'] = 'Awesome'
response['Another-header'] = 'Bad ass'
return response
return wrapper
urlpatterns += patterns('',
url(r'^static/(?P<path>.*)$', custom_headers(serve_static)),
)
If you want your manage.py to have the --nostatic option on by default, you can put this in your manage.py:
if '--nostatic' not in sys.argv:
sys.argv.append('--nostatic')
I think since you are hosting your project on heroku , the generally accepted practice seems to be using S3 for serving static files.
Have a look at this question :
Proper way to handle static files and templates for Django on Heroku
I am not too sure about this but I am sure you should be able to change headers while serving files from S3 atleast this SO question seems to suggest that way
Is it possible to change headers on an S3 object without downloading the entire object?
Hope this helps

How to use django-cumulus for serving Static files?

I'm trying to use django-cumulus for serving files off Rackspace CloudFiles. I'm currently only trying it on my local dev server, using Django 1.4.2.
I can use cumulus's syncstatic management command to upload all my static assets successfully, but I can't seem to display them on my site with the same settings.
If my relevant settings are:
STATIC_URL = '/static/'
CUMULUS = {
'USERNAME': 'myusername',
'API_KEY': 'myapikey',
'CONTAINER': 'mycontainername',
'STATIC_CONTAINER': 'mycontainername',
}
DEFAULT_FILE_STORAGE = 'cumulus.storage.CloudFilesStorage'
STATICFILES_STORAGE = 'cumulus.storage.CloudFilesStaticStorage'
then when I run syncstatic all my apps' static files are uploaded into /mycontainername/static/, as I'd expect. But when I load a page in admin it ignores STATIC_URL and tries to serve assets from URLs like http://uniquekey....r82.cf2.rackcdn.com/path/to/file.css rather than http://uniquekey....r82.cf2.rackcdn.com/static/path/to/file.css.
Also, I can't see how to have my public (non-admin) pages use the static files on CloudFiles, rather than serving them from a local /static/ directory.
Have I missed some crucial setting, or am I doing something else wrong?
I had the same problem. What i did was to
git clone https://github.com/richleland/django-cumulus.git
edit context_processors.py
from django.conf import settings
from cumulus.storage import CloudFilesStorage
def cdn_url(request):
"""
A context processor to expose the full cdn url in templates.
"""
cloudfiles_storage = CloudFilesStorage()
static_url = '/'
container_url = cloudfiles_storage._get_container_url()
cdn_url = container_url + static_url
print {'CDN_URL': cdn_url}
return {'CDN_URL': cdn_url}
Once you are done, install it with sudo python setup.py install
Do note that context_processors.py from django cumulus is actually quite slow