I've been looking into combining static files and serving them as one file. But how would I go about doing this when my django stack is on a apache proxy with a nginx loadbalancer?
regards
Bjarni I.
You might want to look into nginx's try_files directive. Write a view in your Django code that will compress the files and put them onto your nginx server (or something else it can access, like an NFS share), and then have nginx try_files for the version it has first, before falling back to your Django file cruncher if the file doesn't exist yet (at which point nginx would serve it next time).
You might try django-compress. In your settings module, you define groups of static files (js, css), that compress will, well, compress into one file per group. There's then a couple of template tags that you use to include the compressed files in your templates.
For example
#settings.py
COMPRESS_CSS = {
'main': {
'source_filenames': ('css/960gs.css',
'css/main.css',
),
'output_filename': 'css/main.min.r?.css',
'extra_context': {
'media': 'screen,projection',
},
},
# other CSS groups goes here
}
Then, somewhere in your templates (most likely your base template), you use {% compressed_css 'main' %}.
You can define multiple groups if you have some css/js that you want to include in all pages, but some you only want to include in certain pages, or whatever you need. It's pretty flexible.
You can also easily turn compression off for easy debugging.
It also adds a unique version number to the resulting compressed file(s) to get around browser caching problem. This is the r? portion of output_filename.
Related
When I run python manage.py collectstatic, it makes a copy of each image, JavaScript, and CSS file with a hash in the filename:
Post-processed 'css/theme.css' as 'css/theme.afeb1fc222a9.css'
Post-processed 'css/custom.css' as 'css/custom.585e1b29ff9a.css'
...
I'm assuming this is just a way of making a versioned filename for better caching; the client or CDN can be told to cache this file indefinitely, because if I make a change, the hash will differ, and I'll just reference the new version by the new name.
However, I'm not clear on how I'm supposed to reference this URL. The documentation on serving static files just says,
In your templates, either hardcode the url like /static/my_app/example.jpg or, preferably, use the static template tag to build the URL for the given relative path by using the configured STATICFILES_STORAGE storage (this makes it much easier when you want to switch to a content delivery network (CDN) for serving static files).
I went through my templates and dutifully switched every static resource (including the CSS files) from a hardcoded URL to a {% static "..." %} template tag, assuming it would map to the versioned filename where appropriate. But it doesn't.
I'm also using WhiteNoise for serving the resources, and I'm not entirely sure how it affects things, but it also says,
Want forever-cacheable files and compression support? Just add this to your settings.py: STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
But I have that in my settings file and it also doesn't seem to do anything with these versioned filenames.
If DEBUG is True then the static url will be 'css/theme.css' instead of
'css/theme.afeb1fc222a9.css'
Basically, I've a webserver, where I stated in my nginx conf, to show every .cpp as plain text - but I want to make a syntax highlight for more readability.
Any idea how could I proceed?
I want to use google highlights, so any idea about how to insert before an html file before and after every .cpp would suffice.
I thought and tried in the far past using header and footer tags in nginx conf, with no luck whatsoever.
Thanks in advance!
cheers!
As was already pointed out, Nginx is not quite suitable for generating HTML documents by itself. Usually this is a job for a server-side processing language like PHP or Perl. However, there are several ways of solving the problem solely with Nginx.
The first obvious choice would be to use a server-side processing language from within Nginx. There are at least three optional modules for three different languages (Perl, Lua and a dialect of Javascript) that could be used for that.
The problem with this approach is that these modules are rarely available by default, and in many cases you will have to build Nginx manually to enable any of them. Sometimes it can be painful, because as soon as you get your own custom build of Nginx, you will have to support and upgrade it yourself.
There is, however, another option, which involves SSI. It might not be the prettiest solution but it will work. And unlike above-mentioned modules, the SSI support comes with almost every distribution of Nginx. My bet is, your Nginx can do SSI out of the box, without having to compile anything.
So, the configuration goes like this:
# Define a special virtual location for your cpp files
location ~* \.(cpp|h)$ {
# Unless a GET parameter 'raw' is set with 'yes'
if ($arg_raw = 'yes') {
break;
}
# Redirect all the requests for *.cpp and *.h files to another location #js
try_files #js #js;
}
location #js {
ssi on; # Enable SSI in this location
default_type text/html; # Tell the browser that what is returned is HTML
# Generate a suitable HTML document with an SSI insertion
return 200 '<!DOCTYPE html>
<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.9.0/styles/default.min.css">
<script src="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.9.0/highlight.min.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<pre><code class="cpp"><!--# include virtual="$uri?raw=yes" --></code></pre>';
}
Now here is what happens if you request some *.cpp file in your browser:
The request goes to the first location, because the URI ends with cpp.
Then it is redirected to the second location #js, because there is no GET parameter raw in your request.
In the second location the SSI template is generated with return and then immediately processed by the SSI engine because of ssi on.
The include virtual="$uri?raw=yes" tells the SSI engine to make another request (subrequest) from within Nginx to the originally requested file (the internal variable $uri stores the original URI, that is the web path to your cpp file). The difference between the request from your browser and the subrequest made by Nginx is ?raw=yes.
The subrequest again is handled by the first location, but it never goes to the second one, because of the raw GET parameter. In this case the raw contents of the cpp file is returned as a response to the subrequest.
The SSI engine combines this response with the rest of the template and returns the result to the browser. Additionally, default_type tells the browser to render the result as an HTML document.
You can see an example of the output here. I used this highlighting library for this example. You can change it with whatever you prefer simply modifying the SSI template.
In a Django project, I would like to include a static file in a template. The following is the the project structure:
proj/
main/
static/
main/
js/
main.js
news/
templates/
news/
news.html
In news.html, I would like to include the main.js with the following hypothetical format:
{% load staticfiles %}
...
{% include {% static 'main/js/main.js' %} %}
How do I do it?
The include directive only searches file in template directories. What you could do (although I wouldn't) is to change your settings to also include the static files:
TEMPLATE_DIRS = [
...
"path/to/your/statics",
]
I have a couple of ideas.
The easiest thing to do would be to ensure you have the filesystem loader enabled, and you include the relevant directory containing static files in TEMPLATES[0]['DIRS'] (previously TEMPLATE_DIRS). The caveats are that it will not pick up static files contained inside your installed applications automatically, unless you list them in DIRS, too. Another option would be to use STATIC_ROOT, but that will only work if you run collectstatic every time you change the static file, which you normally don't have to do in development. Another pitfall is that it will only work with local static storage, not if you use any CDN or otherwise host your static files on a different host / network / whatever.
The other thing you can do is to write your own template loader which will use the static files API to load statics as templates. This is a bit more involved, but it should be able to access static files regardless of how they are stored and served.
Should you choose either option, you still have to be careful. For instance, you'll have to ensure that the static files you're including as templates are safe to include into HTML or whatever other context you're using them in. There will be no escaping going on.
In case you're trying to optimize the number of requests the client has to make, there are better ways. You're most likely better off implementing some pipeline that will pre-process and minify your static files. Including any significant chunk of CSS / JS into the HTML will make it impossible for clients to take advantage of caching, forcing them to re-download the same static content over and over again, likely impacting the performance negatively.
Edit: Since you want to just include some dynamic JavaScript variables, that's not a static file at all. What you really want is to create a template containing JavaScript code, and not mess with handling static files as templates. There is no rule that would say every javascript needs to be a static file. If it's dynamic, it's not a static file.
For the record, this was a typical instance of the XY problem. Problem X was dynamically assigning values to JavaScript variables when rendering templates. You came up with including static files in templates as a potential solution, but then got stuck with problem Y, which was that you didn't know how to include static files in templates.
Often, I encounter scenarios where I see it makes sense to use template tags in CSS and JavaScript files, such as the use of {{ STATIC_URL }} in CSS to access image. I understand the only way to achieve this is to have CSS and JavaScript files served by Django. I am interested in this approach. But before I commit, I want to hear you experts' experience on it. What are the pros and cons of this approach? Thx.
Pros:
You can make a lot of per-request decisions about how things look and behave.
You can keep the number of different CSS/JS files to a minimum.
Cons:
Browsers tend to cache CSS and JS aggressively, so you'll need to use some aggressive anti-cache techniques. Of course, this means disabling caching for some/all static files.
Every CSS and JS request will consume another thread of your WSGI server. In a normal request/response cycle, each request generally takes up one thread; you're effectively tripling this, at least, so now your app that could handle 200 simultaneous requests now can only handle 66.
When your site makes it big, a CDN probably can't help you.
Alternatives:
Tweak the CSS via javascript, and set a javascript variable inside your page template to control the tweaks.
Use multiple CSS files and control their inclusion dynamically.
Generate static files as needed, but then cache them to disk and serve them via mod_xsendfile. This only works if you are serving static files from somewhere the django process can write to, such as on the same machine or a network mount.
Personally, I've been sticking with the Django team's advice to make CSS and JavaScript static files, served directly by the server instead of via Django. It hasn't been a problem and has simplified a lot of things. Generally, any time I think I need a dynamioc CSS or JS file, there's a way to refactor so I don't.
For example:
the use of {{ STATIC_URL }} in CSS to access image
I'm not sure how variable your {{ STATIC_URL }} is, but I've found that using the <base> tag in my pages fixes a lot of things. I assume this is for background images? Could you update your question to give an example?
Another thing I've done is, if my JavaScript needs dynamic data, I'll put most of the code in a JavaScript library I serve as a static file and then put the minimum dynamic stuff in a <script> tag at the end of the page. Usually I'll put it all in an object (looking a lot like JSON) and then just pass that object to a function. Come to think of it, you could just take all the dynamic stuff, make a dictionary out of it in your view function, encode it into JSON, and pass it via context. Then your page template just looks something like:
<html><head>
...
<script src="{{ STATIC_URL }}/js/foo.js"></script>
...
</head><body>
...
<script>
foo_main({{ foo_params_json|safe }});
</script>
</body></html>
This makes it a lot easier to reuse this code.
I want to serve files dynamically from behind the root of my web directory. Currently I'm serving all the files statically which I did just in order to include my javascripts and css files. I realize this is probably dumb but I'm not able to serve files any other way!
So, I want to serve files dynamically, without the user being able access at another time just by using a url.
My settings
ADMIN_MEDIA_PREFIX = '/media/'
MEDIA_ROOT = os.path.join( APP_DIR, 'site_media' )
MEDIA_URL = 'http://localhost:8000/site_media/'
My URLS
( r'^site_media/(?P<path>.*)$', 'django.views.static.serve', { 'document_root': settings.MEDIA_ROOT } )
Thanks!
What do you mean by "dynamically"? What is it that you're trying to achieve? Do you want to control access to the files?
If you want to run some Django-view that decides whether the user is allowed access to the file or not, you can use sendfile. See the following snippet for code on how to set the correct headers: http://djangosnippets.org/snippets/2226/
Your webserver also needs to support this header, at least apache and nginx which I've worked with does.
You also need to think about how to store and distribute the files as the server that is running the webserver needs access to the files. This will depend on your setup.
You could obfuscate the URLs in such a way that it'd be pretty hard to guess; this isn't at all secure but it would stop the casual poker-arounder. You'd also need to ensure that whatever you use to serve static media in production isn't set to display directory listings.
If you need something a little more secure, here's what I've done in the past:
In models.py:
my_storage = FileSystemStorage(location=settings.MY_FILES_ROOT, base_url='DO_NOT_USE')
class Resource (models.Model):
//...snip...
resource_file = models.FileField(storage = my_storage)
where settings.MY_FILES_ROOT is a path outside of where you keep your static files normally.
Then, in views.py:
def get_file(request, whatever_id):
//check if the user is allowed to access the file here
resource = Resource.objects.get(id=resid)
response = HttpResponse(resource.resource_file.read(), mimetype='text/whatever')
response['Content-Disposition'] = 'attachment; filename=whatever.ext'
return response
Of course, this approach only works if you know the MIME type and file extension of the file, whether it's always the same (the app I did this in always served PDFs) or you extract it from the file.