nginx dynamic server_name with variable - regex

Dear StackOverflow community I am working with a third party that does not support dynamic GET requests (eg example.com?variable=somethingDynamic) thus I resotred to using custom sub-domains, however I prefer not to make a sub domain for each and every dynamic request so I have been wondering:
how can I write server_name in a way to catch two or three dynamic variables?
here is my example server block:
server {
listen 80;
server_name someSecretUrl_$variable1_$variable2.example.com;
root /usr/share/campagins/campagin1;
client_max_body_size 10000m;
proxy_connect_timeout 30000;
location /funnel_webhooks/test {
return 200;
}
location / {
if ($request_method = 'OPTIONS') {
# Tell client that this pre-flight info is valid for 20 days
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
add_header 'Access-Control-Allow-Origin' "$http_origin" always;
add_header 'Access-Control-Allow-Credentials' 'true' always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With,Etag,Last-Modified,HTTP_IF_MODIFIED_SINCE,HTTP_IF_NONE_MATCH' always;
return 204;
}
add_header 'Access-Control-Allow-Origin' "$http_origin" always;
add_header 'Access-Control-Allow-Credentials' 'true' always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With,Etag,Last-Modified,HTTP_IF_MODIFIED_SINCE,HTTP_IF_NONE_MATCH,ETag,Retry-After' always;
add_header 'Access-Control-Expose-Headers' 'ETag,Retry-After' always;
add_header 'Cache-Control' "must-revalidate, post-check=0, pre-check=0" always;
rewrite ^(.*)$ $1?preMadeDataParsers=$variable1&preMadeDataParsersOnResponse=$variable2&$args break;
proxy_buffering off;
proxy_pass http://localhost:3000; #proxy_set_header X-Real-IP $remote_addr;
proxy_set_header REMOTE_ADDR $remote_addr;
}
}
Looking at the code above you will notice I am trying to move $variable1 into a GET variable, and $variable2 into another get variable aswell, how can I achieve such a thing?
thanks!

I wouldn't do this in nginx, I'd do this in your application. (Especially if you expect to expand on this resource.)
I would configure the server to listen on the IP with no virtual hosts at all, so that it answers any request made to the IP. Just leave out the server_name directive:
server {
listen 1.2.3.4:80;
...
Then configure your DNS with a wildcard entry so that *.example.com points to that IP. Now you can hit any_string.example.com and it will resolve to your IP, get answered by the main server block, and passed to your app.
Then, inside your app, look at what hostname was requested. (In PHP for example, this is available via $_SERVER['HTTP_HOST'].) If your app determines that the requested hostname is invalid, just issue a 404 and exit. Otherwise, decode the hostname and process the request.
This way, you can add new variables and new features without editing the nginx config. You could even encode your variables in JSON then BASE64 encode them:
$vars = [
'var1' => 'one',
'var2' => 'two',
'var3' => 'three',
];
$url = base64_encode(json_encode($vars));
eyJ2YXIxIjoib25lIiwidmFyMiI6InR3byIsInZhcjMiOiJ0aHJlZSJ9.example.com
Now you can pass any number of variables, with any names, including indexed and associative arrays. (Though note there is a limit to the domain name length, and you'll have to do something about the + and / characters which I'm pretty sure aren't valid in domain names.)

Related

NGINX filter requests to lan IP

I have received several requests via NGINX that appear to be to my LAN IP 192.168.0.1 as follows:
nginx.vhost.access.log:
192.227.134.73 - - [29/Jul/2021:10:33:47 +0000] "POST /GponForm/diag_Form?style/ HTTP/1.1" 400 154 "-" "curl/7.3.2"
and from Django:
Invalid HTTP_HOST header: '192.168.0.1:443'. You may need to add '192.168.0.1' to ALLOWED_HOSTS.
My NGINX configurations as follows:
upstream django_server {
server 127.0.0.1:8000;
}
# Catch all requests with an invalid HOST header
server {
server_name "";
listen 80;
return 301 https://backoffice.example.com$request_uri;
}
server {
listen 80;
# Redirect www to https
server_name www.backoffice.example.com;
modsecurity on;
modsecurity_rules_file /some_directory/nginx/modsec/modsec_includes.conf;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains;" always;
add_header X-Frame-Options "deny" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
#add_header Content-Security-Policy "script-src 'self' https://example.com https://backoffice.example.com https://fonts.gstatic.com https://code.jquery.com";
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
return 301 https://backoffice.example.com$request_uri;
}
server {
listen 443 ssl http2;
server_name www.backoffice.example.com backoffice.example.com;
modsecurity on;
modsecurity_rules_file /some_directory/nginx/modsec/modsec_includes.conf;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains;" always;
add_header X-Frame-Options "deny" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
#add_header Content-Security-Policy "script-src 'self' https://example.com https://backoffice.example.com https://fonts.gstatic.com https://code.jquery.com";
add_header Referrer-Policy "strict-origin-when-cross-origin";
ssl_certificate /etc/ssl/nginx-ssl/backofficebundle.crt;
ssl_certificate_key /etc/ssl/nginx-ssl/backoffice.key;
access_log /some_directory/nginx/nginx.vhost.access.log;
error_log /some_directory/nginx/nginx.vhost.error.log;
location / {
proxy_pass http://localhost:8000;
proxy_pass_header Server;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_set_header REMOTE_ADDR $remote_addr;
}
location /media/ {
alias /some_directory/backoffice/media/;
}
location /static/ {
alias /some_directory/backoffice/static/;
}
}
My questions:
Is there any way of configuring NGINX to block requests to all LAN IP's?
Can this be done better by ModSecurity?
Is there any way of configuring NGINX to block requests to all LAN IP's?
There is, just make nginx listen only on the public IP (e.g. listen backoffice.example.com:443 ssl http2;). Although I have no idea why you'd want this…
Because if it's an internal IP it cannot be accessed externally (by definition – otherwise you wouldn't call it internal). If that would be the case you'd have more like a problem with your network/firewall.
Regarding the nginx access log, I cannot spot any problem. 192.227.134.73 is not a private IP.
Regarding the Django log, curl -H "Host: 192.168.0.1:443" https://backoffice.example.com would have caused such a request. The "Host" header is just a header after all that can contain anything.

How to return Django project's error messages when I use nginx in production mode

I have developed the Django project and deployed it to the Amazon's free tier EC2 services. Everything is fine except errors message that are not returning back. I am using the project in production mode.
Explanation for above image [Console Log]:
Successful request and response - it was made for existing url
Second request is made intentionaly to non existing url and did not receive any response.
I want to get at least 404 response, the problem I have is not having any response from server. When I run it on server I saw it is logging the results to the server.
Question:
How to return the response that Django is generating when something is wrong.
Extra Info: Those error messages and response are getting generated in djangorestframework's built in template.
Extra Details:
Let me know if I am missing anything.
Brain does really interesting things when it is tired. Thanks to #iklinac. He was right, better I would have used django-cors-headers correctly. It was already installed and working on heroku, when I moved to amazon aws I thought anything was related to NGINX.
Notes to take.
pip install django-cors-headers
Make sure it is in your installed apps.
INSTALLED_APPS = [
...
'corsheaders',
...
]
You will also need to add a middleware class to listen in on responses: # which I have missed
MIDDLEWARE = [ # Or MIDDLEWARE_CLASSES on Django < 1.10
...
'corsheaders.middleware.CorsMiddleware',
'django.middleware.common.CommonMiddleware',
...
]
A list of origins that are authorized to make cross-site HTTP requests
CORS_ORIGIN_WHITELIST = [
"https://example.com",
"https://sub.example.com",
"http://localhost:8080",
"http://127.0.0.1:9000"
]
Then there are few other things you can tweak and use as you want.
Eventually I have changed my nginx.conf to following
upstream hello_django {
server web:8000;
}
server {
listen 80;
location / {
proxy_pass http://hello_django;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
location /staticfiles/ {
alias /home/app/web/staticfiles/;
}
location /mediafiles/ {
alias /home/app/web/mediafiles/;
}
}
Happy coding.)
credits to testdriven.io and django-cors-headers
You are proxy passing requests and they don't get add_header correctly, you should proxy pass after you add headers
location / {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, PATCH, PUT, DELETE';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Headers' 'Authorization,Accept,Origin,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, PATCH, PUT, DELETE';
#
# Custom headers and headers various browsers *should* be OK with but aren't
#
add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range, Authorization';
#
# Tell client that this pre-flight info is valid for 20 days
#
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain; charset=utf-8';
add_header 'Content-Length' 0;
return 204;
}
proxy_pass http://app;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
Other way around would be to add django-cors-headers into your application

Access to image at [Img Link] from origin [domain] has been blocked by CORS policy

I have a domain, domain.com where I have hosted my VueJs application and I have a domain api.domain.com where I have hosted my Django API. I have hosted both of these applications on the AWS (EC2 ubuntu instance).
I am getting this Image CORS error while I am sending add_header 'Access-Control-Allow-Origin' '*' always; in the NGINX.conf
Access to image at 'http://api.deshpardesh.in/media/newspapers/images_qseNU9o.jpeg' from origin 'http://deshpardesh.in' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Here is the actual error.
There were two issues for me.
One is that nginx only processes the last add_header it spots down a tree. So if you have an add_header in the server context, then another in the location nested context, it will only process the add_header directive inside the location context. Only the deepest context.
From the NGINX docs on add_header:
There could be several add_header directives. These directives are inherited from the previous level if and only if there are no add_header directives defined on the current level.
Second issue was that the location / {} block I had in place was actually sending nginx to the other location ~* (.php)$ block (because it would repath all requests through index.php, and that actually makes nginx process this php block). So, my add_header directives inside the first location directive were useless, and it started working after I put all the directives I needed inside the php location directive.
Finally, here's my working configuration to allow CORS in the context of an MVC framework called Laravel (you could change this easily to fit any PHP framework that has index.php as a single entry point for all requests).
server {
root /path/to/app/public;
index index.php;
server_name test.dev;
# redirection to index.php
location / {
try_files $uri $uri/ /index.php?$query_string;
}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
location ~ \.php$ {
fastcgi_split_path_info ^(.+\.php)(/.+)$;
# NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
# With php5-fpm:
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_index index.php;
include fastcgi_params;
# cors configuration
# whitelist of allowed domains, via a regular expression
# if ($http_origin ~* (http://localhost(:[0-9]+)?)) {
if ($http_origin ~* .*) { # yeah, for local development. tailor your regex as needed
set $cors "true";
}
# apparently, the following three if statements create a flag for "compound conditions"
if ($request_method = OPTIONS) {
set $cors "${cors}options";
}
if ($request_method = GET) {
set $cors "${cors}get";
}
if ($request_method = POST) {
set $cors "${cors}post";
}
# now process the flag
if ($cors = 'trueget') {
add_header 'Access-Control-Allow-Origin' "$http_origin";
add_header 'Access-Control-Allow-Credentials' 'true';
}
if ($cors = 'truepost') {
add_header 'Access-Control-Allow-Origin' "$http_origin";
add_header 'Access-Control-Allow-Credentials' 'true';
}
if ($cors = 'trueoptions') {
add_header 'Access-Control-Allow-Origin' "$http_origin";
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Max-Age' 1728000; # cache preflight value for 20 days
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'Authorization,Content-Type,Accept,Origin,User-Agent,DNT,Cache-Control,X-Mx-ReqToken,Keep-Alive,X-Requested-With,If-Modified-Since';
add_header 'Content-Length' 0;
add_header 'Content-Type' 'text/plain charset=UTF-8';
return 204;
}
}
error_log /var/log/nginx/test.dev.error.log;
access_log /var/log/nginx/test.dev.access.log;
}
The gist for the above is at: https://gist.github.com/adityamenon/6753574

logout not working, caching on nginx, how to allow logout?

I have everything cached, if I logged into my account, you will not be able to log out any more) how do you get out when you quit? i need to know how to delete cookies and session! when i'll logout!
P.S. if i'll disable caching on nginx level, everything works fine,
problem in nginx
nginx conf
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
proxy_connect_timeout 5;
proxy_send_timeout 10;
proxy_read_timeout 10;
proxy_buffering on;
proxy_buffer_size 16k;
proxy_buffers 24 16k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
proxy_temp_path /tmp/nginx/proxy_temp;
add_header X-Cache-Status $upstream_cache_status;
proxy_cache_path /tmp/nginx/cache levels=1:2 keys_zone=first_zone:100m;
proxy_cache one;
proxy_cache_valid any 30d;
proxy_cache_key $scheme$proxy_host$request_uri$cookie_US;
server conf
upstream some site {
server unix:/webapps/some/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name server name;
expires 7d;
client_max_body_size 4G;
access_log /webapps/some/logs/nginx-access.log;
error_log /webapps/some/logs/nginx-error.log;
error_log /webapps/some/logs/nginx-crit-error.log crit;
error_log /webapps/some/logs/nginx-debug.log debug;
location /static/ {
alias /webapps/some/static/;
}
location /media/ {
alias /webapps/some/media/;
}
location ~* ^(?!/media).*.(?:jpg|jpeg|gif|png|ico|cur|gz|svg|svgz|mp4|ogg|ogv|webm|htc)$ {
root root_path;
expires 7d;
add_header Pragma public;
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
access_log off;
}
location ~* ^(?!/static).*.(?:css|js|html)$ {
root root_path;
expires 7d;
add_header Pragma public;
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
access_log off;
}
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_cache one;
proxy_cache_min_uses 1;
proxy_cache_use_stale error timeout;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-Proto https;
proxy_set_header Host $http_host;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass http://some;
break;
}
}
error_page 404 /404.html;
location = /error_404.html {
root /webapps/some/src/templates;
}
error_page 500 502 503 504 /500.html;
location = /error_500.html {
root /webapps/some/src/templates;
}
}
Instead of logging out with a GET request, change your logout view to accept a form POST.
POST requests should not be cached.
This has the added security benefit of preventing users from being logged out with iframes or malicious links (ie: https://example.com/logout/, assuming you have not disabled django's CSRF protection).
Note: there is a ticket on django's bug tracker related to this issue.
You have the following question:
i need to know how to delete cookies and session! when i'll logout!
With the following code:
proxy_cache_key $scheme$proxy_host$request_uri$cookie_US;
We first have to know what's in $cookie_US?
If it's simply the name of the login, then you need to realise that anyone who knows the name of the login, and sets their own cookie as such, and knows the complete URL of a hidden resource that such user (and only such user) has access to, and which has been accessed recently (thus freshly cached), can now gain ‘unauthorised’ access to the given resource, since it'll be served straight from cache, and likely without any sort of re-validation.
Basically, for caching user-specific content, you have to make sure that you set http://nginx.org/r/proxy_cache_key to represent an actually secret non-guessable value, which could then be cleared on the user's end to logout. Subsequently, if the user does logout, then your cache is still subject to the replay attacks by anyone who somehow still posesses such secret value, but it'd usually be minimised by a short expiration time of the cache, plus, the secret is still supposed to stay a secret even after logout.
And clearing the session is as easy as simply re-setting the variable to something that wouldn't be giving the access to the user, e.g., you can even implement the whole logout thing entirely within nginx, too:
proxy_cache_key $scheme$proxy_host$request_uri$cookie_US;
location /logout {
add_header Set-Cookie "US=empty; Expires=Tue, 19-Jan-2038 03:14:07 GMT; Path=/";
return 200 "You've been logged out!";
}
P.S. Note that above code technically opens you up to XSS attacks — any other page can simply embed an iframe with /logout on your site, and your users would be logged out. Ideally, you might want to use a confirmation of logout, or check $http_referer to ensure the link is clicked from your own site.

How do i configure the django rest framework pagination url

when I get an object in django rest framework the urls always come absolute with localhost, but in production im going through a proxy on nginx, is there a way to set this url in the settings
Example
count: 11
next: "http://localhost:8000/api/accounts/?ordering=-date_registered&page=2"
previous: null
I need it to be
count: 11
next: "http:/example.com/api/accounts/?ordering=-date_registered&page=2"
previous: null
---------- edit --------------------------
please see my complete nginx config
server {
listen 80;
server_name 123.123.123.123;
root /home/admin/www/site-web/dist;
index index.html;
charset utf-8;
location /static/ {
alias /home/admin/www/site/static/;
}
location /media/ {
alias /home/admin/www/site/media/;
}
location /nginx_status/ {
# Turn on nginx stats
stub_status on;
# I do not need logs for stats
access_log off;
# Security: Only allow access from 192.168.1.100 IP #
# allow 192.168.1.100;
# Send rest of the world to /dev/null #
# deny all;
}
location / {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
try_files $uri $uri/ /index.html;
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' '*';
#
# Om nom nom cookies
#
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
#
# Custom headers and headers various browsers *should* be OK with but aren't
#
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
#
# Tell client that this pre-flight info is valid for 20 days
#
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 204;
}
if ($request_method = 'POST') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
}
if ($request_method = 'GET') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
}
}
location /docs/ {
proxy_pass http://127.0.0.1:8000/docs/;
break;
}
location /api/ {
underscores_in_headers on;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://127.0.0.1:8000/api/;
break;
}
location /admin/ {
proxy_pass http://127.0.0.1:8000/admin/;
break;
}
}
==== super edit====
Sorry guys, i had 'underscores_in_headers on;' i removed it and all is working
================
It sounds like your Host header is not being set properly, which would be an issue in your nginx configuration. The issue is that your Host header that is being sent includes the port number, so Django is including the port number when building out urls. This will cause future issues with CSRF, because CSRF checks do strict port checking when you are not debugging.
This is known to cause issues with SSL for similar reasons.
You can fix this by setting the Host header within Nginx to not include the proxied port.
proxy_set_header Host $http_host;
Note that I used the $http_host variable instead of $host or $host:$server_port. This will ensure that Django will still respect CSRF requests on non-standard ports, while still giving you the correct absolute urls.
Set USE_X_FORWARDED_HOST in your settings to True and make sure you pass it along using your web server(proxy) as well.
When django does build_absolute_uri() it calls get_host() - see below in django.http.request:
def get_host(self):
"""Returns the HTTP host using the environment or request headers."""
# We try three options, in order of decreasing preference.
if settings.USE_X_FORWARDED_HOST and (
'HTTP_X_FORWARDED_HOST' in self.META):
host = self.META['HTTP_X_FORWARDED_HOST']
...
See Real life usage of the X-Forwarded-Host header?