I have a Django project which sends automated e-mails with attached pdfs to users. At the moment I just have a normal /media/ folder with the pdf in it which the code points to. This works in development but throws a server error in production.
My question is how do I server media files in production? I have read a lot about it but can't find exactly what I'm after.
I use collectstatic in my production environment which works for static files, I'd expect something similar for media files.
urls
urlpatterns = [
path('admin/', admin.site.urls),
path('', include('page.urls')),
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
settings
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static_files')
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, "static_media")
views.py (that sends the file)
file_path = os.path.join(settings.STATIC_ROOT+'\\pdf\\free_pdf.pdf')
...
msg.attach_file(file_path)
passenger.wsgi
import os
import sys
sys.path.append(os.getcwd())
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'
import django.core.handlers.wsgi
from django.core.wsgi import get_wsgi_application
from whitenoise import WhiteNoise
SCRIPT_NAME = os.getcwd()
SCRIPT_NAME = '' #solution for damn link problem
class PassengerPathInfoFix(object):
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
from urllib.parse import unquote
environ['SCRIPT_NAME'] = SCRIPT_NAME
request_uri = unquote(environ['REQUEST_URI'])
script_name = unquote(environ.get('SCRIPT_NAME', ''))
offset = request_uri.startswith(script_name) and len(environ['SCRIPT_NAME']) or 0
environ['PATH_INFO'] = request_uri[offset:].split('?', 1)[0]
return self.app(environ, start_response)
application = get_wsgi_application()
application = PassengerPathInfoFix(application)
application = WhiteNoise(application, root='/home/mysite/mysite/static_files')
(this code is from when I abandoned the media folder and was trying to just serve it with my static files)
My proj structure:
|project
|__app
|__static
| |__app
| |__style.css
|__media
| |__app
| |__pdfToBeSent
|__static_files (generated by collectstatic)
|__media_files (i created this)
collectstatic also doesn't copy my projects media files into media_files, I have been copying them in there myself (unsure about this).
I have also tried moving the media file to every possible location in my project
I am also serving my static files with whitenoise.
Thank you.
Since I was in a similar situation (on the same hosting, A2Hosting), I will try to share my understanding of the problem and the solution I opted for.
Pardon me if I may seem presumptuous in this presentation, I'm simply trying to retrace all the points that represent the flow of thoughts that led me to this solution.
A small premise: if with "media files" you intend multimedial files, such as images and so on, I think you shouldn't use the Django media folder as it's designed to serve files uploaded by the users.
Not knowing if your PDFs are indeed uploaded by some user or not, I'll try to expose my solution anyway.
When in a production environment, Django isn't going to serve static and media files.
For the former I too used WhiteNoise, while for the latter the approach is different (at least on a shared hosting base).
When we set Debug = False in settings.py I suspect that the media folder, created along with the Django project, becomes somewhat unreadable/unaccessible (I cannot tell if by the hand of Django or by the hand of the hosting, or a conjuction of the two).
The official method to handle media files is indeed to rely on an external storage service (like Amazon S3), but this solution isn't suitable for budget limited scenarios.
In my situation, I had the Django app running on a subdomain related to my main domain.
Principal domain: www.mydomain.com
App domain: subdomain.mydomain.com
Leaving the media folder created with the Django project where it was, I created another one in the following unix path:
/home/A2hosting_username/subdomain.mydomain.com/media
Then, in settings.py I changed the MEDIA_ROOT variable from:
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
to:
MEDIA_ROOT = '/home/A2hosting_username/subdomain.mydomain.com/media'
After that, in the model.py class used to define the database and interact with it, I specified the path to the uploaded media (videos, in my case) in this way:
class Video(models.Model):
name = models.CharField(max_length=50)
path = models.FileField(upload_to="../media/videos", null=True, verbose_name="")
Since we don't have access to the Apache config file, it may be useful to edit the .htaccess file (relative to subdomain.mydomain.com) in the following way, to prevent the browser to "time out" when the uploaded file is somewhat heavy:
<ifModule mod_headers.c>
Header set Connection keep-alive
</ifModule>
I hope this can be of any help.
Related
I have a Django app that I just added to the already deployed Django web on Apache.
Because it is ran by Apache, path of the media folder seems to be different.
My app lets the user upload an excel file which then changes numbers and save as csv file.
(only showed relevant folders/code snippets)
Current directory
converts\
_init_.py
apps.py
forms.py
models.py
converter.py
urls.py
views.py
main\
settings.py
urls.py
wsgi.py
meida\
excels\
example.xlsx
csvs\
example.csv
static\
manage.py
settings.py
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'
main\urls.py
urlpatterns = [
path('', RedirectView.as_view(url='/converts/')),
path('converts/', include('converts.urls')),
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
The part that causes the problem is the following in converts/converter.py:
def convertExcel(name):
path = 'media/excels/'
key = path + name
wb = load_workbook(key)
Originally in development, a function in view calls convertExcel(example.xlsx) and the workbook, via media/excels/example.xlsx, finds the correct file to work with the loaded workbook. But in production server, it gives
FileNotFoundError: [Errno 2] No such file or directory: 'media/excels/example.xlsx'
My Question is:
Do I have to back track from where apache2.conf is to find the path? Or is there a way to set path to my django project so i can just set path in my convertExcel() as 'media/excels'? Or is there any other way I can call the uploaded workbook?
Any kind of help/comment would be appreciated.
Please comment if additional information is needed.
My guess is that you should use MEDIA_ROOT variable because it points to the uploaded files. So you would have
def convertExcel(name):
from django.conf import settings
path = os.path.join(settings.MEDIA_ROOT, 'excels')
key = os.path.join(path, name)
wb = load_workbook(key)
I am trying to serve dynamically created files on my Django server.
The code works fine and serves the image file from media folder on my local server.
It also works and serves the image file in Production server but only when I set debug to True in Prod server.
But when I set debug to False in Prod Server, I get a file not found error (in console):
GET http://thesitename/media/imgs/img.png 404 (Not Found)
The server serves the files from the /static folder, but does not serve the files from /media folder.
my settings.py looks like this:
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, "static/")
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, "frontend/media/")
urls.py looks like this:
app_name = 'frontend'
urlpatterns = [ ###urls###
]+ static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
Views.py looks like this:
def template_page(request, loation, par, date):
### generating image using pil here ###
pil_image.save("frontend/media/imgs/img.png”)
path_to_image_passing_to_template = “/media/imgs/img.png”
params = {
'path_to_image_passing_to_template' : path_to_image_passing_to_template
}
return render(request, 'frontend/template_page.html', params)
html looks like this:
<img src="{{path_to_image_passing_to_template}}" />
and file structure is like this:
/backend
urls.py
views.py
/frontend
/static
/media
/imgs
/templates
/frontend
urls.py
views.py
/thesite
settings.py
urls.py
wsgi.py
/static
I don't understand why it should work with debugging turned on in prod server and not otherwise!!!
I have looked at several questions but did not find any query close to this kind of issue. I took some help while writing working out the dynamic media files code from this answer earlier but am stuck right now.
Confirm that you have 'os.environ' module loaded...try executing the
code below on the production server.
import os
print(os.environ.get('MEDIA_ROOT'))
Make sure it shows the expected path.
then ensure that your 'urls.py' and 'Views.py' have 'import os' line appearing at the top of the file. Good Luck!
To integrate Django and Ember, I have decided to serve my Ember SPA in a Django view (avoids CORS issues, only one server for frontend and API, etc). I do it like this:
# urls.py
urlpatterns = [
url(r'^admin/', include(admin.site.urls)),
url(r'^api/', include(api_urls, namespace='api')),
...
url(r'^$', views.emberapp, name='emberapp'),
...
]
# views.py
from django.http import HttpResponse
def emberapp(request):
# The Ember frontend SPA index file
# This only works in development, and is anyway hacky
EMBER_FE_INDEX_HTML = '/absolute/path/to/my/frontend/static/fe-dist/index.html'
template_file = open(EMBER_FE_INDEX_HTML)
html_content = index_file.read()
index_file.close()
return HttpResponse(html_content)
The index.html is part of the static assets. In development this is very easy:
The index.html is directly accessible to the Django application in the file system
I know the absolute path to the index file
But in production things are more complex, because the static assets are not local to the django application, but accessible on Amazon S3. I use django-storages for that.
How can I read the contents of a static file from a view, in a generic way, no matter what backend is used to store/serve the static files?
First, I don't think the way you do it is a good idea.
But, to answer your question: In your settings.py, you likely have defined the directory where Django will collect all static files.
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
So in your view, you just need to fetch the file os.path.join(settings.STATIC_ROOT, 'index.html')
That said, you should serve index.html via the webserver, same as your static/ files, robots.txt, favicon.ico, etc. Not through Django. The webserver is much faster, uses proper caching, and its just one line in your Nginx or Apache settings, instead of an entire view function in Django.
This is my current solution. Works in development, not sure about production yet (it is a pain that you need to commit untested code to verify production-related code in Heroku)
from django.conf import settings
from django.http import HttpResponse
from django.core.files.storage import get_storage_class
FE_INDEX_HTML = 'fe/index.html' # relative to the collectstatic directory
def emberapp(request):
# The Ember frontend SPA index file
# By getting the storage_class like this, we guarantee that this will work
# no matter what backend is used for serving static files
# Which means, this will work both in development and production
# Make sure to run collectstatic (even in development)
# TODO: how to use this in development without being forced to run collectstatic?
storage_class = get_storage_class(settings.STATICFILES_STORAGE)
# TODO: reading from a storage backend can be slow if assets are in a third-party server (like Amazon S3)
# Maybe streaming the static file from the server would be faster?
# No redirect to the Amazon S3 asset, please, since the Ember App needs to
# run from the same URL as the API, otherwise you get CORS issues
with storage_class().open(FE_VWORKS_INDEX_HTML) as index_file:
html_content = index_file.read()
return HttpResponse(html_content)
Or, to reply with an StreamingHttpResponse, which does not force Django to read the whole file in memory (and wait for it to be read):
def emberapp(request):
# The Ember frontend SPA index file
# By getting the storage_class like this, we guarantee that this will work
# no matter what backend is used for serving static files
# Which means, this will work both in development and production
# Make sure to run collectstatic (even in development)
# TODO: how to use this in development without being forced to run collectstatic?
storage_class = get_storage_class(settings.STATICFILES_STORAGE)
index_file = storage_class().open(FE_INDEX_HTML)
return StreamingHttpResponse(index_file)
I'm trying to optimize my webpage and am having trouble setting expiration date headers on my static files.
I am running django-1.5, python-2.7.3.
Here's my cache settings in settings.pyso far:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
'LOCATION': os.path.join(PROJECT_ROOT, 'cache/'),
}
}
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_SECONDS = 5 * 60
CACHE_MIDDLEWARE_KEY_PREFIX = ''
MIDDLEWARE_CLASSES = (
'django.middleware.cache.UpdateCacheMiddleware',
...
'django.middleware.cache.FetchFromCacheMiddleware',
)
And my static file settings in settings.py:
import os.path
PROJECT_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_ROOT = os.path.abspath(os.path.join(PROJECT_DIR, '..'))
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'staticfiles/')
STATIC_URL = '/static/'
STATICFILES_DIRS = (
os.path.join(PROJECT_DIR, 'static'),
)
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
# 'django.contrib.staticfiles.finders.DefaultStorageFinder',
)
The closest advice I've found was here, but I'm unable to modify the .htaccess files on Heroku.
Any help is greatly appreciated. Thanks!
The django staticfiles app does not provide out of the box support for custom headers. You'll have to hack together your own view to serve up the files and add custom headers to the HttpResponse.
But You should not be serving your static files using Django. This is a terrible idea.
Django is single-threaded, and blocking. So every time you're serving a user a static file, you're literally serving nothing else (including your application code, which is what Django is there for).
Django's staticviews file is insecure, and unstable. The documentation specifically says not to use it in production. So do not use it in production. Ever.
In production you shouldn't serve static files with Django. See the warning boxes on this page: https://docs.djangoproject.com/en/1.4/ref/contrib/staticfiles/#static-file-development-view
In development, Django's contrib.staticfiles app automatically serves staticfiles for you by overwriting the runserver command. This way you can't control the way it serves the static files.
You can prevent the staticfiles app from serving the static files by adding the --nostatic option to the runserver command:
./manage.py runserver --nostatic
Then you can write an url config to manually serve the static files with headers you want:
from functools import wraps
from django.conf import settings
from django.contrib.staticfiles.views import serve as serve_static
from django.conf.urls import patterns, url
urlpatterns = patterns('', )
if settings.DEBUG:
def custom_headers(view_func):
#wraps(view_func)
def wrapper(request, *args, **kwargs):
response = view_func(request, *args, **kwargs)
response['Custom-header'] = 'Awesome'
response['Another-header'] = 'Bad ass'
return response
return wrapper
urlpatterns += patterns('',
url(r'^static/(?P<path>.*)$', custom_headers(serve_static)),
)
If you want your manage.py to have the --nostatic option on by default, you can put this in your manage.py:
if '--nostatic' not in sys.argv:
sys.argv.append('--nostatic')
I think since you are hosting your project on heroku , the generally accepted practice seems to be using S3 for serving static files.
Have a look at this question :
Proper way to handle static files and templates for Django on Heroku
I am not too sure about this but I am sure you should be able to change headers while serving files from S3 atleast this SO question seems to suggest that way
Is it possible to change headers on an S3 object without downloading the entire object?
Hope this helps
I'm trying to use django-cumulus for serving files off Rackspace CloudFiles. I'm currently only trying it on my local dev server, using Django 1.4.2.
I can use cumulus's syncstatic management command to upload all my static assets successfully, but I can't seem to display them on my site with the same settings.
If my relevant settings are:
STATIC_URL = '/static/'
CUMULUS = {
'USERNAME': 'myusername',
'API_KEY': 'myapikey',
'CONTAINER': 'mycontainername',
'STATIC_CONTAINER': 'mycontainername',
}
DEFAULT_FILE_STORAGE = 'cumulus.storage.CloudFilesStorage'
STATICFILES_STORAGE = 'cumulus.storage.CloudFilesStaticStorage'
then when I run syncstatic all my apps' static files are uploaded into /mycontainername/static/, as I'd expect. But when I load a page in admin it ignores STATIC_URL and tries to serve assets from URLs like http://uniquekey....r82.cf2.rackcdn.com/path/to/file.css rather than http://uniquekey....r82.cf2.rackcdn.com/static/path/to/file.css.
Also, I can't see how to have my public (non-admin) pages use the static files on CloudFiles, rather than serving them from a local /static/ directory.
Have I missed some crucial setting, or am I doing something else wrong?
I had the same problem. What i did was to
git clone https://github.com/richleland/django-cumulus.git
edit context_processors.py
from django.conf import settings
from cumulus.storage import CloudFilesStorage
def cdn_url(request):
"""
A context processor to expose the full cdn url in templates.
"""
cloudfiles_storage = CloudFilesStorage()
static_url = '/'
container_url = cloudfiles_storage._get_container_url()
cdn_url = container_url + static_url
print {'CDN_URL': cdn_url}
return {'CDN_URL': cdn_url}
Once you are done, install it with sudo python setup.py install
Do note that context_processors.py from django cumulus is actually quite slow