I have Nginx running on an Amazon Linux EC2 instance. It is listening for connections to https://dcv01.example.com and then using proxy_pass with a wildcard cert to serve the DCV client from an internal IP address. The page displays properly with no SSL errors. However, instead of the login prompt, I get a red message that says "Failed to communicate with server.".
I attempted to use the allowed-http-host-regex with the value ^(www\.)?dcv01\.example\.com$, but then I get a 403. I also tried allowed-ws-origin-regex, but it does nothing different (still get Failed to communicate...)
What do I need to do to get DCV working behind a reverse proxy?
Here is the server block in my Nginx config:
server {
listen 443 ssl;
server_name dcv01.example.com;
include wildcard.conf; //just has ssl_certificate and ssl_certificate_key directives
location / {
proxy_redirect off;
proxy_pass $scheme://10.0.10.131:8443;
}
}
Related
I have a custom Django web app sitting behind an NGINX proxy server.
I am seeing occasional errors come through from my Django server with messages like
Invalid HTTP_HOST header: 'my.domain.com:not-a-port-number'. The domain name provided is not valid according to RFC 1034/1035.
where the part after the colon has been a variety of wack garbage like,
my.domain.com:§port§/api/jsonws/invoke
my.domain.com:xrcr0x4a/?non-numeric-port-bypass
my.domain.com:80#+xxxxtestxxxx/
my.domain.com:80#+${12311231}{{12311231}}/
I am able to replicate a similar error using curl where the request url and the host header are different:
curl https://my.domain.com --header 'Host: my.domain.com:not-a-port-number'
I suspect that these are likely coming from our network security scanning software trying to find vulnerabilities in our custom web apps, but I am a little surprised that NGINX allows these requests to make it through to my Django server, especially if, as the 500 error output suggests, these are invalidly formatted headers.
Trying to prepare for the worst, is there anything I should change or be concerned about with this for the sake of security? Is there a best practice for this situation that I am unaware of? Should NGINX be filtering out these sorts of requests?
For my own convenience it would be nice to not to see the noise of these 500 errors coming from Django while I am on the lookout for real app level errors, but simply hiding the alerts seems to be a poor solution.
Additional Info:
I have ALLOWED_HOSTS set to 'my.domain.com' in my Django settings.py file.
NGINX configuration:
server {
listen 80 default_server;
return 444;
}
server {
listen 80;
server_name my.domain.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl default_server;
ssl_certificate /path/to/cert.cabundle.pem;
ssl_certificate_key /path/to/cert.key;
return 444;
}
# App https server.
# Serve static files and pass requests for application urls to gunicorn socket.
server {
listen 443 ssl;
server_name my.domain.com;
ssl_certificate /path/to/cert.cabundle.pem;
ssl_certificate_key /path/to/cert.key;
...
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://unix:/path/to/gunicorn.sock;
}
}
I never met such a situation on practice, however I can guess nginx select your last server block to handle the request according to the information from the SNI Client Hello extension which by some reason (malformed request from scanning software?) is different from the Host header value in the encrypted request.
If you want to filter those requests at the nginx level, you can try this:
server {
listen 443 ssl;
server_name my.domain.com;
ssl_certificate /path/to/cert.cabundle.pem;
ssl_certificate_key /path/to/cert.key;
if ($http_host != my.domain.com) {
return 444;
}
...
}
PS. For the security considerations I never use my real certificate in a stub server block like this one:
server {
listen 443 ssl default_server;
ssl_certificate /path/to/cert.cabundle.pem;
ssl_certificate_key /path/to/cert.key;
return 444;
}
Use a self-signed one as shown here. The reason if obvious - dear hacker, if you don't know what exactly domain you are scanning now, I'm not going to tell you what it is.
I am building an application which is right now working on localhost. I have my entire dockerized application up and running at https://localhost/.
HTTP request is being redirected to HTTPS
My nginx configuration in docker-compose.yml is handling all the requests as it should.
I want my application accessible from anywhere hence i tried using Ngrok to route the request to my localhost. Actually i have a mobile app in development so need a local server for apis.
Now, when i enter ngrok's url like abc123.ngrok.io in the browser, the nginx converts it to https://localhost/. That works for my host system's browser, as my web app is working there only, but when i open the same in my mobile emulator. It doesn't work.
I am newbie to nginx. Any suggestions will be welcomed.
Here's my nginx configuration.
nginx.conf
upstream web {
ip_hash;
server web:443;
}
# Redirect all HTTP requests to HTTPS
server {
listen 80;
server_name localhost;
return 301 https://$server_name$request_uri;
}
# for https requests
server {
# Pass request to the web container
location / {
proxy_pass https://web/;
}
location /static/ {
root /var/www/mysite/;
}
listen 443 ssl;
server_name localhost;
# SSL properties
# (http://nginx.org/en/docs/http/configuring_https_servers.html)
ssl_certificate /etc/nginx/conf.d/certs/localhost.crt;
ssl_certificate_key /etc/nginx/conf.d/certs/localhost.key;
root /usr/share/nginx/html;
add_header Strict-Transport-Security "max-age=31536000" always;
}
This configuration i got from a tutorial.
First of all, you set redirection from every HTTP request to HTTPS:
# Redirect all HTTP requests to HTTPS
server {
listen 80;
server_name localhost;
return 301 https://$server_name$request_uri;
}
You are using $server_name variable here, so every /some/path?request_string HTTP request to your app would be redirected to https://localhost/some/path?request_string. At least change the return directive to
return 301 https://$host$request_uri;
Check this question for information about difference between $host and $server_name variables.
If these are your only server blocks in your nginx config, you can safely remove the server_name localhost; directive at all, those blocks still remains the default blocks for all incoming requests on 80 and 443 TCP ports.
The second one, if you are using self-signed certificate for localhost be ready for browser complains about mismatched certificate (issued for localhost, appeared at abc123.ngrok.io). If it doesn't break your mobile app, its ok, but if it is, you can get the certificate for your abc123.ngrok.io domain from Lets Encrypt for free after you start your ngrok connection, check this page for available ACME clients and options. Or you can disable HTTPS at all if it isn't strictly requred for your debug process, just use this single server block:
server {
listen 80;
# Pass request to the web container
location / {
proxy_pass https://web/;
}
location /static/ {
root /var/www/mysite/;
}
}
Of course this should not be used in production, only for debugging.
And the last one. I don't see any sense encrypting traffic between nginx and web containers inside docker itself, especially if you already setup HTTP-to-HTTPS redirection with nginx. It gives you nothing in the terms of security but only some extra overhead. Use plain HTTP protocol on port 80 for communications between nginx and web container:
upstream web {
ip_hash;
server web:80;
}
server {
...
location / {
proxy_pass http://web;
}
}
I'm trying to configure Certbot (Letsencrypt) with Nginx.
I get this error :
- The following errors were reported by the server:
Domain: koomancomputing.com
Type: unauthorized
Detail: Invalid response from
http://koomancomputing.com/.well-known/acme-challenge/xvDuo8MqaKvUhdDMjE3FFbnP1fqbp9R66ah5_uLdaZk
[2600:3c03::f03c:92ff:fefb:794b]: "<html>\r\n<head><title>404 Not
Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
Not Found</h1></center>\r\n<hr><center>"
Domain: www.koomancomputing.com
Type: unauthorized
Detail: Invalid response from
http://www.koomancomputing.com/.well-known/acme-challenge/T8GQaufb9qhKIRAva-_3IPfdu6qsDeN5wQPafS0mKNA
[2600:3c03::f03c:92ff:fefb:794b]: "<html>\r\n<head><title>404 Not
Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
Not Found</h1></center>\r\n<hr><center>"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.
in /etc/nginx/sites-available/koomancomputing :
server {
listen 80;
server_name koomancomputing.com www.koomancomputing.com;
location = /favicon.ico { access_log off; log_not_found off; }
location /staticfiles/ {
root /home/kwaku/koomancomputing;
}
location /media/ {
root /home/kwaku/koomancomputing;
}
location / {
include proxy_params;
proxy_pass http://unix:/run/gunicorn.sock;
}
}
my DNS A/AAAA records :
I didn't know what to do, so I did a search and find django-letsencrypt app, but I don't know hot to use :
Your domain has a proper AAAA record configured to your server over IPv6, and certbot chose that to validate your server.
However, your server block as configured under nginx only listens to port 80 on IPv4 for your domain. When certbot requests Let's Encrypt to access your challenge and issue a certificate, nginx isn't configured to properly respond with the challenge on IPv6. It often in this case returns other things (such as a 404 in your case, or a default site).
You can resolve this by modifying the first two lines to also listen on all IPv6 addresses for your server:
server {
listen 80;
listen [::]:80;
# other configuration
}
After editing, restart nginx and run certbot again.
Your Nginx server is responding with a 404 error because it does not define a route to /.well-known needed by certbot to verify challenges. You need to modify the Nginx config file to tell it how to respond to certbot's challenges.
Certbot can update the Nginx config file for you.
First, make sure your config file is enabled. Run sudo service nginx reload and check for the presence of a file called /etc/nginx/sites-enabled/koomancomputing.
Then, run certbot --nginx -d koomancomputing.com -d www.koomancomputing.com
The --nginx flag tells certbot to find an Nginx config file with a matching server name and update that file with SSL info.
server {
listen 80;
listen [::]:80;
# other configuration
}
Works for both IPV4 and IPV6 after adding this restart nginx.
For me, it worked after I removed and installed the latest certbot version using snapd.
I use cloudflare proxy option and it failed for certbot 0.31.0.
After installing certbot 1.27 and configuring the cert newly, it works fine even proxy toggle is on in cloudflare.
I was setting up nginx on my aws ubuntu instance. At first every time went well, but after I config nginx and try to connect django, I can't even see the welcome page from either public ip nor the localhost(which was able to access from both sides). The nginx status command shows nginx is running.
Here's my nginx config:
/nginx/sites-available/mysite.com
server{
charset utf-8;
listen 80;
server_name my_aws_ip;
location /static{
alias my_django_static_path;
}
location / {
proxy_set_header Host $host;
proxy_pass http://unix:tmp/mysite.socket;
}
}
And I made a link to /nginx/sites-enabled/
It appears that every time I restarted nginx, I will be able to see the welcome page. However, after that, nginx refuses connections.
I didn't change anything in nginx.conf. Do I need to?
server_name should be your domain name, IP address should be specified as part of the listen directive
proxy_pass http://unix:tmp/mysite.socket;
Not sure where you are hoping this will end up, but you need to decide if you are sending it via http or to a socket. Not both. Having said that if it's for django then it's not proxy_pass you want at all
I'm guessing you mean:
uwsgi_pass unix:/tmp/mysite.socket;
You'll also need to include these somewhere in your config
I am trying to set up proxy_protocol in my nginx config. My server sits behind an AWS load balancer (ELB), and I have enabled Proxy Protocol on that for both ports 80 and 443.
However, this is what I get when I hit my server:
broken header: "��/��
'���\DW�Vc�A{����
�#��kj98���=5���g#32ED�</A
" while reading PROXY protocol, client: 172.31.12.223, server: 0.0.0.0:443
That is a direct copy paste from the nginx error log - wonky characters and all.
Here is a snip from my nginx config:
server {
listen 80 proxy_protocol;
set_real_ip_from 172.31.0.0/20; # Coming from ELB
real_ip_header proxy_protocol;
return 301 https://$http_host$request_uri;
}
server {
listen 443 ssl proxy_protocol;
server_name *.....com
ssl_certificate /etc/ssl/<....>;
ssl_certificate_key /etc/ssl/<....?;
ssl_prefer_server_ciphers On;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!DSS:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4;
ssl_session_cache shared:SSL:10m;
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
ssl_stapling on;
ssl_stapling_verify on;
...
I can't find any help online about this issue. Other people have had broken header issues, but the error with bad headers are always readable - they don't look like they are encoded like they are for me.
Any ideas?
Two suggestions:
Verify that your ELB listener is configured to use TCP as the protocol, not HTTP. I have an LB config like the following that's routing to Nginx with proxy_protocol configured:
{
"LoadBalancerName": "my-lb",
"Listeners": [
{
"Protocol": "TCP",
"LoadBalancerPort": 80,
"InstanceProtocol": "TCP",
"InstancePort": 80
}
],
"AvailabilityZones": [
"us-east-1a",
"us-east-1b",
"us-east-1d",
"us-east-1e"
],
"SecurityGroups": [
"sg-mysg"
]
}
You mentioned that you have enabled Proxy Protocol in the ELB, so I'm assuming you've followed AWS setup steps. If so then the ELB should be crafting the HTTP request correctly with the first line as something like PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n. However if the HTTP request is not coming into Nginx with the PROXY ... line then it could cause the problem you're seeing. You could reproduce that if you hit the EC2 DNS name directly in the browser, or you ssh into the EC2 instance and try something like curl localhost, then you should see a similar broken header error in the Nginx logs.
To find out whether it works with a correctly formed HTTP request you can use telnet:
$ telnet localhost 80
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80
GET /index.html HTTP/1.1
Host: your-nginx-config-server_name
Connection: Keep-Alive
Then check the Nginx logs and see if you have the same broken header error. If not then the ELB is likely not sending the properly formatted PROXY request, and I'd suggest re-doing the ELB Proxy Protocol configuration, maybe with a new LB, to verify it's set up correctly.
I had similar situation, nginx had 'proxy_protocol' on but AWS ELB settings was not on, so I got the similar message.
Solutions to edit settings to turn it on:
I had this error and came across this ticket:
https://trac.nginx.org/nginx/ticket/886
which ultimately led me to figuring out that I had an unneeded proxy_protocol declaration in my nginx.conf file. I removed that and everything was working again.
Oddly enough, everything worked fine with nginx version 1.8.0, but when I upgraded to nginx version 1.8.1 is when I started seeing the error.
I got this unreadable header issue too and here are the cause and how I fixed it.
In my case, Nginx is configured with use-proxy-protocol=true properly. It complains about the broken header solely because AWS ELB did not add the required header (e.g. PROXY TCP4 198.51.100.22 203.0.113.7 35646 80) at all. Nginx sees the encrypted HTTPS payload directly. That's why it prints out all the unreadable characters.
So, why didn't the AWS ELB add the PROXY header? It turned out I used wrong ports in the commands to enable Proxy Protocol policy. Instance ports should be used instead of 80 and 443.
The ELB has the following port mapping.
80 -> 30440
443 -> 31772
The commands should be
aws elb set-load-balancer-policies-for-backend-server \
--load-balancer-name a19235ee9945011e9ac720a6c9a49806 \
--instance-port 30440 \
--policy-names ars-ProxyProtocol-policy
aws elb set-load-balancer-policies-for-backend-server \
--load-balancer-name a19235ee9945011e9ac720a6c9a49806 \
--instance-port 31272 \
--policy-names ars-ProxyProtocol-policy
but I used 80 and 443 by mistake.
Hope this helps somebody.
Stephen Karger's solution above is correct, you must adjust make sure to configure your ELB to support proxy. Here is the AWS docs for doing exactly that: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html. The docs are a bit daunting at first, so if you want you can just skip to steps 3 and 4 under the Enable Proxy Protocol Using the AWS CLI section. Those are the only necessary steps for enabling the proxy channeling. Additionally, as Stephen also suggested, you must make sure that your ELB is using TCP instead of http or https, as both of these will not behave properly with ELB's proxy implementation. I suggest moving your socket channel away from common ports like 80 and 443, just so you can still maintain those standardized connections with their default behavior. Of course, making that call is entirely dependent on how your app stack looks.
If it helps, you can use the npm package wscat to debug your websocket connections like so:
$ npm install -g wscat
$ wscat --connect 127.0.0.1
If the connection works in local, then it is for sure your load balancer. However, if it doesn't, there is almost definitely a problem with your socket host.
Additionally, a tool like nmap will aid you in discovering open ports. A nice checklist for debugging:
npm install -g wscat
# can you connect to it from within the server?
ssh ubuntu#69.69.69.69
wscat -c 127.0.0.1:80
# can you connect to it from outside the server?
exit
wscat -c 69.69.69.69:80
# if not, is your socket port open for business?
nmap 69.69.69.69:80
You can also use nmap from within your server to discover open ports. to install nmap on ubuntu, simply sudo apt-get install nmap. on osx, brew install nmap
Here is a working config that i have, although it does not provide ssl support at the moment. In this configuration, I have port 80 feeding my rails app, port 81 feeding a socket connection through my elb, and port 82 open for internal socket connections. Hope this helps somebody!! Anybody using Rails, unicorn, and Faye to deploy should find this helpful. :) happy hacking!
# sets up deployed ruby on rails server
upstream unicorn {
server unix:/path/to/unicorn/unicorn.sock fail_timeout=0;
}
# sets up Faye socket
upstream rack_upstream {
server 127.0.0.1:9292;
}
# sets port 80 to proxy to rails app
server {
listen 80 default_server;
keepalive_timeout 300;
client_max_body_size 4G;
root /path/to/rails/public;
try_files $uri/index.html $uri.html $uri #unicorn;
location #unicorn {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded_Proto $scheme;
proxy_redirect off;
proxy_pass http://unicorn;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}
error_page 500 502 503 504 /500.html;
location = /500.html {
root /path/to/rails/public;
}
}
# open 81 to load balancers (external socket connection)
server {
listen 81 proxy_protocol;
server_name _;
charset UTF-8;
location / {
proxy_pass http://rack_upstream;
proxy_redirect off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
# open 82 to internal network (internal socket connections)
server {
listen 82;
server_name _;
charset UTF-8;
location / {
proxy_pass http://rack_upstream;
proxy_redirect off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}