I have some web content behind the follwoing arch, for which I need to serve different content for desktop and mobile clients.
client > AWS Cloudfront > Nginx > ...
cloudfront has the builtin ability to identify the user-agent and they kind of unify it into 4 special headers (for example CloudFront-Is-Desktop-Viewer) which are either true or false.
on Nginx I'm trying to decide which content to serve according to those headers.
for example:
location / {
if ($http_CloudFront-Is-Desktop-Viewer = true) {
proxy_pass http://upstream;
break;
}
root /var/www/static/en-US;
try_files $uri /index.html;
}
so in the above case it should go to the upstream if the CloudFront-Is-Desktop-Viewer value is true, and get the static files from nginx if it's false or not present.
but for some reason I always get the static files from nginx.
I'm sure this header is being forwarded, I've even tried to send it directly from chrome with a header modifier.
what am i missing?
many thanks
Try this variable:
$http_cloudfront_is_desktop_viewer
From manual about embedded variables:
$http_name nginx
arbitrary request header field; the last part of a variable name is
the field name converted to lower case with dashes replaced by
underscores
Related
First of all, google or SO search didn't help me: lots of tips regarding django's staticfiles, which I believe are not relevant here.
I have inherited a project consisting of:
Django backend in form of API returning JSON responses only;
standard Swampdragon deployment pushing realtime updates to frontend; very little configuration has been done here;
Frontend webapp built on Backbone and marionette.js, compiled and minified by Grunt.
My problem is: the frontend needs to know addresses for swampdragon and django servers; right now those values are hardcoded, so there is for example a Backbone model with lines like:
url: function() {
return App.BACKEND_URL+'settings/map';
}
Why hardcoded: backend can be served on any port or have a subdomain to itself; frontend is static and normally would be simply thrown into /var/www (for Apache) or would use some very simple nginx config. Both will be served from the same place, but there is no guarantee the port numbers or subdomains would match.
Idea number 1: try to guess what BACKEND_URL is from javascript, by taking window.location.host and appending standard port. That's hackish and error prone.
Idea number 2: move frontend to Django and make it ask for swampdragon credentials (they would be sent in the context of home view). Problem with that is, the frontend files are compiled by grunt. So where Django would kindly expect something like:
<script src="{% static 'scripts/vendor/modernizr.js' %}"></script>
I actually have
<script src="scripts/vendor/a8bcb0b6.modernizr.js"></script>
Where 'a8bcb0b6' is grunt's hash/version number and will be regenerated during next minification/build. Do I need to add additional logic to get rid of such stuff and copy grunt's output directory to django's static and template dirs?
Or is there another way to make this work, the right one, I am missing?
Your architecture is already clean, no need to make Django know about grunt or serve static files, and no need to use JS hacks to guess port numbers
Reverse Proxy
Use a reverse proxy like nginx or any other web server you like as a front end to both the static files and the REST API.
In computer networks, a reverse proxy is a type of proxy server that
retrieves resources on behalf of a client from one or more servers.
These resources are then returned to the client as though they
originated from the proxy server itself. (Wikipedia)
I will outline the important aspects without going into too much detail:
URL for the REST API
We make configs so that nginx will forward the API requests to Django
location /api {
proxy_pass http://127.0.0.1:8000; # assumes Django listens here
proxy_set_header Host $http_host; # preserve host info
}
So the above assumes your Django REST is mapped to /api and runs on port 8000 (e.g. you can run gunicorn on that port, or any other server you like)
http://nginx.org/en/docs/http/ngx_http_proxy_module.html
URL for our front end app
Next nginx will serve the static files that come out of grunt, by simply pointing it to the static folder
location / { alias /app/static/; }
The above assumes your static resources are in /app/static/ folder (like index.html, your CSS, JS etc). So this is primarily to load your BackboneJS app.
Django static files
Next step is not required, but if you have static files that you use with the Django app (static files that are generated with ./manage.py collectstatic, e.g. the django admin or the UI of Django REST Framework etc), simply map according to your Django settings.py STATIC_URL and STATIC_ROOT
location /static { alias /app/django_static_root/; }
/static and django_static_root being the STATIC_URL and STATIC_ROOT respectively
To sum up
So e.g. when you hit example.com/, nginx simply serves up the static files, then when a JS script makes REST call to /api, it gets trapped in the /api nginx location and gets forwarded to Django
End result is, example.com/ and example.com/api both hit the same front end web server, which proxies them to the right places
So there you have it, reserve proxying solves your ports and subdomain issues (and many others, like slow static files from Django and same-origin policies in web browsers and firewalls not liking anything besides default HTTP and HTTPS ports)
I have a development server setup that does some dynamic rooting to allow me to set up quick test projects by detecting domain and subdomain in server_name and using it to set the root.
server_name ~^(?<subdomain>\w*?)?\.?(?<domain>\w+\.\w+)$;
This works well, allowing me to set the root path based on variables $subdomain and $domain
For a specific type of project though, I also need to be able to further split the subdomain variable into two variables base on if the subdomain contains a dash.
e.g.
mysubdomain should not split but remain as variable $subdomain,
but mysubdomain-tn would be seperated into 2 variable $subdomain and $version
You need to complicate your regular expression a bit more:
server_name ~^(?<subdomain>\w*?)(-(?<version>\w*?)?)?\.?(?<domain>\w+\.\w+)$;
EDIT:
There are several ways to debug an Nginx configuration, including the debugging log, echo module and, in some extreme situations, even using a real debugger. However, in most cases adding custom headers to the response is enough to get the necessary information.
For example, I tested the regular expression above using this simple configuration:
server {
listen 80;
server_name ~^(?<subdomain>\w*?)(-(?<version>\w*?)?)?\.?(?<domain>\w+\.\w+)$;
# Without this line your browser will try to download
# the response as if it were a file
add_header Content-Type text/plain;
# You can name your headers however you like
add_header X-subdomain "$subdomain";
add_header X-domain "$domain";
add_header X-version "$version";
return 200;
}
Then I added domains mydomain.local, mysubdomain-tn.mydomain.local and mysubdomain-tn.mydomain.local to my hosts file, opened them in a browser with open debug panel (F12 in most browsers) and got the results.
I'm currently developing a site where the functionality needs to be split into separate subdomains, dashboard.example.com, admin.example.com, and facebook.example.com. I would like everything to be served through a single Django project because everything will be using the same core models. I'm using Nginx as a front-facing proxy server handling static files and passing all other requests to Apache.
The solution I thought of was to map each of these subdomains to the appropriate app through nginx:
server {
listen 80;
server_name dashboard.example.com;
...
location / {
proxy_pass http://127.0.0.1/dashboard/;
...
}
}
server {
listen 80;
server_name admin.example.com;
...
location / {
proxy_pass http://127.0.0.1/admin/;
...
}
}
...doing that for each subdomain, effectively mapping the subdomains to their respective app url namespaces. The problem I encountered was that Django was unaware of the mapping, so when it reversed a URL, it would prepend /dashboard/, etc. to it, creating URLs like dashboard.example.com/dashboard/dashboard/. I figure I could write a custom reverse function to strip out the unnecessary subdirectory, but that seems like a band-aid.
Is there a better way to accomplish what I need, or should I restructure the project?
Thanks for your help.
Django's Sites framework (https://docs.djangoproject.com/en/1.7/ref/contrib/sites/) should be sufficient for this, if not, take a look at django-subdomains (http://django-subdomains.readthedocs.org/en/latest/) as seems to have a means of resolving your reverse URLs (based off a quick Google search, I've never used it myself!)
Specifically, I think I need Nginx to not consume (capture?) a piece of the URI I'm using to route a location. But I don't know if such a thing is possible.
Let me back up. I'm transitioning my app's setup. Before, I had an Nginx config file with a single location block matching everything:
server {
listen 80;
server_name ec2-54-234-175-21.compute-1.amazonaws.com;
location / {
...
proxy_pass http://localhost:8000/;
}
With this setup, up until now, I've just been running a Django app. After Nginx routes the user to the only endpoint, Django consumes the whole URI. What this means is that internally, Django chops off /api/ and then just has 1.0, which it also needs.
Now, I'm going to use Nginx to proxy for multiple servers, each hosting an app. Because of the aforementioned internal routing, the Django app needs to receive (at least) a URI of /1.0, while the other (on Flask) needs to get /api/2.0. However, in order for the Nginx location directives to make any sense, I have to differentiate the two applications. What I've worked out is that I'll just have the following two location directives:
server {
listen 80;
server_name ec2-54-234-175-21.compute-1.amazonaws.com;
location /api/[1.0] {
...
proxy_pass http://localhost:8000/;
}
location / {
...
proxy_pass http://localhost:8080/;
}
However, note the [1.0] in brackets. If that is there, what I've realized is that in order to actually access the intended resource, I have to enter a URI of /api/1.0/1.0. So, somehow I need a non-consuming location in my nginx conf. If I knew how to express this in simpler terms, I would. Is what I want possible?
About a week after asking this question, a coworker pointed out the answer I wanted, and it's very simple. The key is in the malleability that Nginx gives you with its routing. Because I wanted to access a resource that was at /api/1.0, but still needed to differentiate the two apps in my config, I do the following:
server {
listen 80;
server_name ec2-54-234-175-21.compute-1.amazonaws.com;
location /api/1.0 {
...
proxy_pass http://localhost:8000/api/1.0;
}
location /api/2.0 {
...
proxy_pass http://localhost:8080/api/2.0;
}
This effectively makes the URL "non-consuming", as I wrote above, because I use the desired resource URL to route with, and then duplicate it in referencing the actual location within the specific app.
Maybe this will help someone else.
Disclaimer:
I'm working in a project where exist an "huge" webapp that have an api for mobiles, so change the api is not an option.
This application was developed time ago and several developers have worked on it,
Having said that, the problem is this;
In the api for mobile of this site (just views than returns json data), the code is looking for a token but does in the headers of request:
token = request.META.get('HTTP_TOKEN')
When I test this api locally, works fine, but in production doesn't, so, I try to figure out whats going on and found this:
django converts headers, even custom headers to keys in request.META, I use urllib2 and requests for test the api and the problem in production is that in production server the request.META never has a key called HTTP_TOKEN, so, doing a little of debug I seriously think the problem is the way we serve the django application.
We are using django1.3, nginx, gunicorn, virtualenvwrapper, python2.7.
My prime suspect is nginx, I think, in someway nginx receive the header but don' forward it to django, I try to do some research about this, but I only found infor about security headers and custom headers from nginx, but I dont find doc or something about how to tell nginx that allows that header and don't remove it.
I need help here, the first thing is test if nginx receives the header, but I just know a little about nginx and I don't know how to tell it to log the headers of requests.
Thanks
Update
nginx conf file
If Django is accessed using uwsgi_pass, then in the appropriate location(s) ...
# All request headers should be passed on by default
# Make sure "Token" response header is passed to user
uwsgi_pass_header Token;
If Django is accessed using fastcgi_pass, then in the appropriate location(s) ...
# All request headers should be passed on by default
# Make sure "Token" response header is passed to user
fastcgi_pass_header Token;
If Django is accessed using proxy_pass, then in the appropriate location(s) ...
# All request headers should be passed on by default
# but we can make sure "Token" request header is passed to Django
proxy_set_header Token $http_token;
# Make sure "Token" response header is passed to user
proxy_pass_header Token;
These should help eliminate the possibility that Nginx is not passing things along from your issue.
In your nginx configuration file (f.e. mysite_nginx.conf) in the server section add this parameter: uwsgi_pass_request_headers on;.
For example:
server {
# the port your site will be served on
listen 8000;
...
underscores_in_headers on;
}
And if access to Django goes through uwsgi_pass, you need to add this one parameter uwsgi_pass_request_headers on; in location section.
For example:
location / {
include /etc/nginx/uwsgi_params; # the uwsgi_params file you installed
uwsgi_pass_request_headers on;
uwsgi_pass django;
}
I think this is what you need:
log_format combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" "$http_http_token" "$upstream_http_http_token"'
to log what is happening.
You might look deeper into the proxy_set_header section on the upstream proxy module to see how to pass on the headers you need.
You can find the documentation here:
http://wiki.nginx.org/HttpLogModule
http://wiki.nginx.org/NginxHttpUpstreamModule
http://wiki.nginx.org/NginxHttpProxyModule#proxy_set_header
http://wiki.nginx.org/HttpProxyModule#proxy_pass_request_headers
The last entry seems to indicate that nginx passes most headers by default
I didn't find a real answer, but was able to make a workaround. I was having the same problem with RFC standard headers if-none-match and if-modified-since, so my solution is tested for those headers.
Added to my nginx config:
uwsgi_param HTTP_IF_NONE_MATCH $http_if_none_match;
uwsgi_param HTTP_IF_MODIFIED_SINCE $http_if_modified_since;
I cannot explain why nginx refuses to pass these headers to uwsgi by default. This config forces it. Pages generate 304s as appropriate now.
For the original question about the non-standard "token" header, this should do the trick:
uwsgi_param HTTP_TOKEN $http_token;
The answers above are enough for us to figure out the way. But there is still an annoying point we need to know. Yes, it is very annoying.
proxy_set_header X_FORWARDED_FOR # oops it never works. 1.16.1 on centos7
proxy_set_header X-FORWARDED-FOR # this will do the job
So you get it. Underscore could never appear in the customized variable name. Use a hyphen instead.
Maybe Nginx uses underscores in some grammar cases. Someone pointed out the official reference will be appreciated.
It depends on how the custom header is named. My was in format "SomethingLike.this", it contains a dot. It was not possible to rename the header in the request, because it is not our code. So writing this would not work:
proxy_set_header SomethingLike.this $http_somethinglike.this;
proxy_pass_header SomethingLike.this;
Also this would not work:
underscores_in_headers on;
because I would need dots_in_headers directive which does not exist.
But I found I can pass ALL headers simply by adding:
ignore_invalid_headers off;
It may be insecure to pass all headers, please use with caution.