I'm a nooob devops guy in town. I've a nodejs app running on amazon instance pointing to DNS's.
Route53 screenshot
Problem
when I try to reach the subdomain, I get
curl -i http://site.soothsayer.co/
curl: (7) Failed connect to site.soothsayer.co:80; Connection timed out
or
$ curl -i http://site.soothsayer.co/
HTTP/1.1 502 Bad Gateway
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8
Via: 1.1 Comverse 6.2.23
Content-Length: 287
<!DOCTYPE html PUbliC "-//W3C//Dtd XHTML basic 1.0//EN"
"http://www.w3.org/tr/1999/REC-html401-19991224/loose.dtd">
<html>
<head>
<title>Error</title>
</head>
<body leftmargin="0" topmargin="0" style="font-family:arial">
Error: Requested url content cannot be resolved.
</body>
</html>
Debugging
nslookup result is
Server: 127.0.1.1
Address: 127.0.1.1#53
Non-authoritative answer:
site.soothsayer.co canonical name = internal-p-site-us-east-1-1077850846.us-east-1.elb.amazonaws.com.
Name: internal-p-site-us-east-1-1077850846.us-east-1.elb.amazonaws.com
Address: 172.31.46.66
I can see proper html response from GET http://localhost:3000 or GET http://localhost:8085 (configured in nginx) within the amazon machine.
I'm using nginx/1.4.7 as HTTP and reverse proxy server (as per my understanding), conf file being
$ vi /etc/nginx/conf.d/site.soothsayer.co.conf
# the IP(s) on which your node server is running. I chose port 3000.
upstream site.soothsayer.co {
server 127.0.0.1:3000;
}
# the nginx server instance
server {
listen 0.0.0.0:8085;
server_name site-proxy.soothsayer.co site-proxy.soothsayer.co;
access_log /var/log/nginx/site-proxy.soothsayer.co.log;
# pass the request to the node.js server with the correct headers and much more can be added, see nginx config options
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://site.soothsayer.co/;
proxy_redirect off;
}
}
My /etc/resolv.conf is,
; generated by /sbin/dhclient-script
search ec2.internal
nameserver 172.31.0.2
iptables of amazon machine,
[root#ip-172-31-42-77 rof]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Security Group on amazon is default,
Custom TCP Rule TCP 8086 0.0.0.0/0
I don't see any logs at /var/log/nginx/error.log. This is what my little brain could debug. I want to fix this issue.
References
nginx, dns - domain.com resolves but subdomain.domain.com doesn't
AWS - elastic load balancer unable to handle the request
amazon ec2 instance unable to resolve host
Curl amazon EC2 instance
So, issue was with load balancer being internal. Made it internet-facing. Everything looks Majestic now.
References
Internet-facing and Internal Load Balancers
not sure, but your nginx conf with listen 0.0.0.0:8085 may block any remote access.
Try :
listen 8085;
Related
I'm running Django on Gunicorn behind a NGINX reverse proxy, and an AWS Application Load Balancer. The ALB has 2 listeners. The HTTPS listener forwards to a Target Group in port 80, and the HTTP listener redirects to HTTPS.
The ALB thus connects with an AWS ECS Cluster, which runs a task with 2 containers: one for the Django app, and another for the NGINX that acts as a reverse proxy for the Django app. Here's the configuration for the NGINX reverse proxy:
upstream django {
server web:8000;
}
server {
listen 80;
listen [::]:80;
location / {
proxy_pass http://django;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_ssl_server_name on;
proxy_redirect off;
}
}
This configuration ensures that whenever the client tries to hit the website app using an HTTP request, he gets redirected to HTTPS. And everything works fine with one exception. In Django, when I run request.is_secure() I'm getting False instead of True as expected. If I run request.build_absolute_uri(), I get http://mywebsite.com and not https://mywebsite.com as expected.
I already tried adding the following lines to settings.py:
USE_X_FORWARDED_HOST = True
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
as explained in the documentation, but it doesn't work. Whenever I inspect request.META (or the raw request.headers), I'm seeing 'HTTP_X_FORWARDED_PROTO': 'http' (and the equivalent raw 'X-Forwarded-Proto': 'http') instead of https as expected. The stack is correctly forwarding 'HTTP_X_FORWARDED_HOST': 'mywebsite.com' from the client, but the scheme is being ignored.
Can anyone help me identify what I'm doing wrong and how to fix it? Thanks
With a Classic ELB you specify the "instance port" (see the listeners tab) and that controls the protocol that you send downstream to nginx. In that scenario it is common to attach an SSL cert to the 443 port but send HTTP down port 80 to nginx. the port 80 listener also sends HTTP. In that setup, where only HTTP is coming in from the load balancer it is your job to inspect the X-Forwarded-Proto header and perform a permanent redirect to HTTPS. That's because the classic ELB could not redirect HTTP to HTTPS. With the Application Load Balancer (ALB) I believe you can redirect HTTP to HTTPS if you want to speak 443 to nginx.
In your specific case it seems like you were only listening on Port 80 so you were probably only sending HTTP from the load balancer. Check you Instance Port and protocol.
I have a load balancer and an Nginx that sits behind the LB.
Below is the nginx config.
upstream app {
server service_discovery_name.local:5005;
}
server { // Reverse proxy for VPC ES to be available on public
listen 80;
location / {
proxy_pass vpc-es-domain-url;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
server { // reverse proxy for django app
listen 8005;
location / {
proxy_pass http://app;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
}
I've a listener attached to the ALB, listening at port 80, that forwards the traffic to the target IP. The target group has the private IP of the Nginx container. I use Fargate-ECS container.
Now when I route to ALB_url:80, it opens up the elasticsearch. However, when I route to ALB_url:8005, it fails to load anything. The django_app is running at port 5005, check by explicitly browsing to the IP=:5005.
I believe the nginx config is right. I want my traffic to be routed via ALB -> Nginx -> apps. What exactly am I missing?
When you configure an ALB you must create a listener, specify the port and the action(forward the request to Target Group or make redirects), you can create a multiples listener using different ports, for example, you can have a listener listening in 80 port and doing redirects to HTTPs and another listener with 443 port forwarding the request to Target Group.
According to that, I understand that your configuration is:
- ALB listening in 80 port and sending the request to Target Group.
- Target Group listening in 80 port and sending the request to Fargate Task(nginx server)
When you route to ALB_URL:80 the request is forwarded to Target Group by 80 port and the request is sending to Fargate task. But when you route to ALB_URL:8005 that will no work because the ALB doesn't have a listener for that port.
You can create a listener with 8005 port that forwards the request to a Target Group listening in the 8005. with this configuration when you route ALB_url:8005 the request will be sent to TG created and then will send to the Fargate task and will take the configuration into the Nginx config.
ALB---> listener 80 ----> Target Groupt port 80 ----> ECS Task Nginx
ALB---> listener 8005 ---> Target Groupt port 8005 ----> ECS Task Nginx
Don't forget to validate the Security groups to allow 8005 port
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#target-group-routing-configuration
I am using an AWS Lightsail Load Balancer in conjunction with another Lightsail ec2 instance so that I can use the free certificate manager built into the Lightsail Load Balancer. This seems to automatically forward all traffic from my Load Balancer to my ec2 nginx server on port 80 so that the following config also supports https connections:
server {
listen 80;
server_name mountainviewwebtech.ca www.mountainviewwebtech.ca;
location / {
proxy_pass http://localhost:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
However, when I add the following lines to ensure that http redirects to https, I receive the error ERR_TOO_MANY_REDIRECTS as my ec2-instance is only receiving traffic on port 80 even when using a secure connection so it just keeps redirecting over and over again.
if ($scheme != https) {
return 301 https://$host$request_uri;
}
Is there anyway to obtain the original $scheme before it was forwarded to my ec2 instance?
If you are wanting a 'free' SSL cert, using Let's Encrypt's CertBot will get you there (and save you money). I have had good success with this option.
Using the load balancer as the SSL terminator ends the secure connection at the load balancer. Thus un-encrypted traffic is sent from the balancer to the compute instances (port 80). If you want to forward SSL traffic to the compute instances, see if the load balancer will do port forwarding. This being Lightsail I doubt it will be possible.
I'm running into '502 Bad Gateway' issues for HTTPS requests when using AWS Elastic Load Balancer (Application type) in front of EC2 instances running Nginx. Nginx is acting as a reverse proxy on each instance for a waitress server serving up a python app (Pyramid framework). I'm trying to use TLS termination at the ELB so that the EC2 instances are only dealing with HTTP. Here's the rough setup:
Client HTTPS request > ELB (listening on 443, forwarding to 80 on backend) > Nginx listening on port 80 (on Ec2 instance) > forwarded to waitress/Pyramid (on same ec2 instance)
When I make requests on HTTPS I get the 502 error. However, when I make regular HTTP requests I get a response as expected (same setup as above except ELB is listening on port 80).
Some additional info:
ELB health checks are working.
All VPC/Security groups are configured correctly (I believe).
I'm using an AWS certificate on the ELB using the standard setup/walkthrough on AWS.
I SSH'd into the Ec2 instance and in the Nginx access log it looks like the HTTPS request are still encrypted? Or some encoding issue?
And here's nginx.conf on the EC2 instance:
#user nobody;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
access_log /etc/nginx/access.log;
sendfile on;
# Configuration containing list of application servers
upstream app_servers {
server 127.0.0.1:6543;
}
server {
listen 80;
server_name [MY-EC2-SERVER-NAME];
# Proxy connections to the application servers
# app_servers
location / {
proxy_pass http://app_servers;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
}
}
Ok I figured it out (I'm a dummy). I had two listeners set up on the ELB, one for 80 and one for 443, which was correct. The listener for 80 was set up correctly to forward to backend (Nginx) port 80 over HTTP as expected. The 443 listener was INCORRECTLY configured to send to port 80 on the backend over HTTPS. I updated the 443 listener to use the same rule as the 80 listener (i.e. listen on 443 but send to backend 80 over HTTP) and it worked. Disregard y'all.
I am trying to follow the configuration of nginx on elastic beanstalk and some things do not add up.
The instance is opening port 80 in the security groups, so I assume all incoming traffic is coming through that port
The nginx configuration in cat /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf states:
server {
listen 8080;
location / {
proxy_pass http://nodejs;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
gzip on;
}
Port 8080? Where did that come from? I've tried to mess with it, this is the actual directive that is working.
server_name is missing, yet tt doesn't matter what you put in it. If I put any value in server_name myself, this server rule will still match all requests, even the ones that are not remotely reseble the server_name value.
While connected to the instance itself it seems both ports are being served:
[ec2-user#ip-172-31-45-222 ~]$ sudo netstat -lnptu
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 22506/nginx
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 22506/nginx
Yet again, 8080 is never opened in the security group, so elastic load balanced is getting inside through port 80. Does traffic magically goes from 80 to 8080? Any ideas what is going on here?
You are forgetting to look at one part of that nginx config:
upstream nodejs {
server 127.0.0.1:8081;
keepalive 256;
}
That part is telling nginx to make a group of servers called nodejs as you can read about here.
8081 is the port that NodeJS is running on (if you use the sample application for instance).
You can verify this by looking at the Elastic Beanstalk logs:
-------------------------------------
/var/log/nodejs/nodejs.log
-------------------------------------
Server running at http://127.0.0.1:8081/
Then if we continue in the nginx.conf file we can see what you already posted:
server {
listen 8080;
location / {
proxy_pass http://nodejs;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
This tells nginx to use the proxy pass module to pass all from port 8080 to our upstream group nodejs which is running on port 8081. This means that port 8081 is just for accessing it locally but port 8080 is what let's outside entities talk to the nginx which then passes stuff onto nodejs.
Some of the reasoning for not exposing NodeJS directly can be found in this StackOverflow answer.
Port 8080 is used because it is the HTTP alternate port that is "commonly used for Web proxy and caching server, or for running a Web server as a non-root user."
That explains the ports. Now the issue of ELB and how things are talking to each other.
Since the security group is only allowing access on port 80, there is an iptables rule that is setup to forward port 80 to port 8080. This allows non-root to bind to port 8080 because lower port numbers require root privileges.
You can verify this by running the following:
[ec2-user#ip-xxx-xx-xx-x ~]$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- anywhere anywhere tcp dpt:http redir ports 8080
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- anywhere anywhere tcp dpt:http redir ports 8080
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
So in summary, when you load your CNAME, the load balancer is rerouting the traffic to a given instance on port 80, which is allowed through the security group, then iptables is forwarding that to port 8080, which is the port that nginx is using a proxy to pass the traffic to port 8081 which is the local port of NodeJS.
Here's a diagram:
incoming connections
-> :80 - Load Balancer
-> :80 - Security group
-> :80 -> :8080 - EC2 instance, iptables forward
-> :8080 -> :8081 - nginx, proxy pass
-> :8081 - nodejs, your app
Hopefully that helps.