Traslate nginx ingress rule with snippet to Istio - istio

I have a nginx ingress controller and expose services with him, we planned change to istio to ingress traffic.
I have a ingress rule that contains snippet:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/server-snippet: |
location ~* "^/" {
proxy_pass "https://127.0.0.1";
proxy_set_header Host $http_x_forwarded_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_connect_timeout 10s;
proxy_send_timeout 120s;
proxy_read_timeout 120s;
client_max_body_size 300m;
}
name: foo
spec:
ingressClassName: bar
rules:
- host: foo.bar
tls:
- hosts:
- foo.bar
This ingress copy http_x_forwarded_host to Host and send to nginx ingress.
There are any idea to convert this rule to istio?
Thanks.
Marco

Welcome to SO!,
It should be doable theoretically with below Istio building components:
Use regex based rewrites
nginx.ingress.kubernetes.io/rewrite-target => EnvoyFilter to HTTP_ROUTE object
(example to be found on github here)
Forward 'X-Forwarded-For/X-Real-IP' headers to upstream host
If your application needs to know the real client IP address use the Gateway Network Topology (Alpha) feature.
Remark:
Attached by you the source manifest file seems to be suffering from a known issue of latest nginx ingress controller, which reveals in following error on my env:
Error from server (BadRequest): error when creating "STDIN": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
-------------------------------------------------------------------------------
Error: exit status 1
2021/06/21 11:05:45 [emerg] 851#851: invalid number of arguments in "proxy_set_header" directive in /tmp/nginx-cfg063051389:453
nginx: [emerg] invalid number of arguments in "proxy_set_header" directive in /tmp/nginx-cfg063051389:453
nginx: configuration file /tmp/nginx-cfg063051389 test failed

Related

Eks ALB Ingress-Controller configure add_header

I currently have a Kubernetes cluster on AWS (EKS).
For the ingress I have an ingress controller deployed.
I have a deployment with a pod in which there are two containers. A PHP Container and an Nginx Container. The Nginx container only acts as a proxy and I would like to remove it.
Currently the nginx .conf has the following that I don't know how to pass it to the ALB ingress.
($ request_method = 'POST') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT, User-Agent, X-Requested-With, If-Modified-Since, Cache-Control, Content-Type, Range';
add_header 'Access-Control-Expose-Headers' 'Content-Length, Content-Range';
I don't know if it is possible to pass the add_header to the ALB Ingress. Does anyone know if it can be done or if on the contrary it is necessary to install an Nginx Ingress Controller?
Thanks
i think this will help your question. https://gitanswer.com/how-to-config-cors-with-alb-go-aws-load-balancer-controller-485142972
because the alb ingress controller only opens an alb that routes traffic to the service it cannot be done like that. and as you said working with nginx ingress controller will solve your problem.

Invalid X-Forwarded-For header with cloudflare and ingress-nginx

I'm having some trouble getting the correct IP headers. I am using the following proxy setup:
Cloudflare -> Amazon NLB -> Ingress-nginx (k8s)
my ingress-nginx has the following config:
config:
use-forwarded-headers: "true"
real-ip-header: "CF-Connecting-IP"
forwarded-for-header: "CF-Connecting-IP"
set-real-ip-from: "0.0.0.0/0"
proxy-buffer-size: "16k"
proxy-buffers-number: "8"
For some reason the x-real-ip header is correct, but the x-forwarded-for header is not:
REMOTE ADDR: 127.0.0.1
X FORWARDED FOR: 10.0.102.38 <- Wrong
X-REAL-IP: xx.xxx.xxx.xxx <- Correct
The ingress-nginx loadbalancer (NLB) has:
External Traffic Policy: Local
as per the doc. And the following annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: 3600
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: *
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: xxxx
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
service.beta.kubernetes.io/aws-load-balancer-type: nlb
To be on the safe side I also enabled ExternalTrafficPolicy: Local on the app's service but to no avail.
Enabling use-proxy-protocol: "true" in the config breaks the app (probably because of CloudFlare).
2021-09-19 12:42:17
" while reading PROXY protocol, client: x.x.x.x, server: 0.0.0.0:80
Any help would be appreciated.
Cloudflare supports the X-Forwarded-For header, so you could try configuring:
config:
[...]
forwarded-for-header: "X-Forwarded-For"
You seem to be able of reading CF-Connecting-IP correctly in your setup, so X-Forwarded-For should also be available in the request coming from Cloudflare.
For more information you can refer to the support article.

Cannot access airflow web server via AWS load balancer HTTPS because airflow redirects me to HTTP

I have an airflow web server configured at EC2, it listens at port 8080.
I have an AWS ALB(application load balancer) in front of the EC2, listen at https 80 (facing internet) and instance target port is facing http 8080.
I cannot surf https://< airflow link > from browser because the airflow web server redirects me to http : //< airflow link >/admin, which the ALB does not listen at.
If I surf https://< airflow link > /admin/airflow/login?next=%2Fadmin%2F from browser, then I see the login page because this link does not redirect me.
My question is how to change airflow so that when surfing https://< airflow link > , airflow web server will redirect me to https:..., not http://.....
so that AWS ALB can process the request.
I tried to change base_url of airflow.cfg from http://localhost:8080 to https://localhost:8080, according to the below answer, but I do not see any difference with my change....
Anyway, how to access https://< airflow link > from ALB?
Since they're using Gunicorn - you can configure the forwarded_allow_ips value as an evironment variable instead of having to use an intermediary proxy like Nginx.
In my case I just set FORWARDED_ALLOW_IPS = * and it's working perfectly fine.
In ECS you can set this in the webserver task configuration if you're using one docker image for all the Airflow tasks (webserver, scheduler, worker, etc.).
Finally I found a solution myself.
I introduced a nginx reverse proxy between ALB and airflow web server: ie.
https request ->ALB:443 ->nginx proxy: 80 ->web server:8080. I make the nginx proxy tell the airflow web server that the original request is https not http by adding a http header "X-Forwarded-Proto https".
The nginx server is co-located with the web server. and I set the config of it as /etc/nginx/sites-enabled/vhost1.conf (see below). Besides, I deletes the /etc/nginx/sites-enabled/default config file.
server {
listen 80;
server_name <domain>;
index index.html index.htm;
location / {
proxy_pass_header Authorization;
proxy_pass http://localhost:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_http_version 1.1;
proxy_redirect off;
proxy_set_header Connection "";
proxy_buffering off;
client_max_body_size 0;
proxy_read_timeout 36000s;
}
}
User user389955 own solution is probably the best approach, but for anyone looking for a quick fix (or want a better idea on what is going on), this seems to be the culprit.
In the following file (python distro may differ):
/usr/local/lib/python3.5/dist-packages/gunicorn/config.py
The following section prevents forwarded for headers from anything other than local
class ForwardedAllowIPS(Setting):
name = "forwarded_allow_ips"
section = "Server Mechanics"
cli = ["--forwarded-allow-ips"]
meta = "STRING"
validator = validate_string_to_list
default = os.environ.get("FORWARDED_ALLOW_IPS", "127.0.0.1")
desc = """\
Front-end's IPs from which allowed to handle set secure headers.
(comma separate).
Set to ``*`` to disable checking of Front-end IPs (useful for setups
where you don't know in advance the IP address of Front-end, but
you still trust the environment).
By default, the value of the ``FORWARDED_ALLOW_IPS`` environment
variable. If it is not defined, the default is ``"127.0.0.1"``.
"""
Changing from 127.0.0.1 to specific IP's or * if IP's unknown will do the trick.
At this point, I haven't found a way to set this parameter from within airflow config itself. If I find a way, will update my answer.
We solved this problem in my team by adding an HTTP listener to our ALB that redirects all HTTP traffic to HTTPS (so we have an HTTP listener AND an HTTPS listener). Our Airflow webserver tasks still listen on port 80 for HTTP traffic, but this HTTP traffic is only in our VPC so we don't care. The connection from browser to the load balancer is always HTTPS or HTTP that gets rerouted to HTTPS and that's what matters.
Here is the Terraform code for the new listener:
resource "aws_lb_listener" "alb_http" {
load_balancer_arn = aws_lb.lb.arn
port = 80
protocol = "HTTP"
default_action {
type = "redirect"
redirect {
port = "443"
protocol = "HTTPS"
status_code = "HTTP_301"
}
}
}
Or if you're an AWS console kinda place here's how you set up the default action for the listener:
Console
I think you have everything working correctly. The redirect you are seeing is expected as the webserver is set to redirect from / to /admin. If you are using curl, you can pass the flag -L / --location to follow redirects and it should bring you to the list of DAGs.
Another good endpoint to test on is https://<airflow domain name>/health (with no trailing slash, or you'll get a 404!). It should return "The server is healthy!".
Be sure you have https:// in the base_url under the webserver section of your airflow config.
Digging into the gunicorn documentation: it seems to be possible to pass any command line argument (when gunicorn command is called) via the GUNICORN_CMD_ARGS environment variable.
So what I'm trying out is setting GUNICORN_CMD_ARGS=--forwarded-allow-ips=* since all the traffic will come to my instance from the AWS ALB... I guess the wildcard could be replaced with the actual IP of the ALB as seen by the instance, but that'll be next step...
Since I'm running on ECS, I'm passing it as:
- Name: GUNICORN_CMD_ARGS
Value: --forwarded-allow-ips=*
in the Environment of my task's container definition.
PS: from the doc, this possibility was added as of gunicorn 19.7; for comparison, Airflow 1.10.9 seems to be on gunicorn 19.10 so good to go with any (more or less) recent version of Airflow !
I encountered this issue too when using the official apache airflow helm chart (version 1.0.0).
Problem
Originally I had configured the webserver service with type LoadBalancer.
webserver:
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-west-2:1234512341234:certificate/231rc-r12c3h-1rch3-1rch3-rc1h3r-1r3ch
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
This resulted in the creation of a classic elastic load balancer.
This mostly worked but when I clicked on the airflow logo (which links to https://my-domain.com), I'd get redirected to http://my-domain.com/home which failed because the load balancer was configured to use HTTPS only.
Solution
I resolved this by installing the AWS Load Balancer Controller on my EKS cluster and then configuring ingress.
The ingress-related portion of the chart config looks like this:
ingress:
enabled: true
web:
host: my-airflow-address.com
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/subnets: subnet-01234,subnet-01235,subnet-01236
alb.ingress.kubernetes.io/scheme: internal # if in private subnets
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
webserver:
service:
type: NodePort
Notes
It might be possible to configure the webserver to use an ALB instead of classic ELB and configure it to handle the HTTP routing, but I have not tested it.

elastic beanstalk weird nginx configuration

I am trying to follow the configuration of nginx on elastic beanstalk and some things do not add up.
The instance is opening port 80 in the security groups, so I assume all incoming traffic is coming through that port
The nginx configuration in cat /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf states:
server {
listen 8080;
location / {
proxy_pass http://nodejs;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
gzip on;
}
Port 8080? Where did that come from? I've tried to mess with it, this is the actual directive that is working.
server_name is missing, yet tt doesn't matter what you put in it. If I put any value in server_name myself, this server rule will still match all requests, even the ones that are not remotely reseble the server_name value.
While connected to the instance itself it seems both ports are being served:
[ec2-user#ip-172-31-45-222 ~]$ sudo netstat -lnptu
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 22506/nginx
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 22506/nginx
Yet again, 8080 is never opened in the security group, so elastic load balanced is getting inside through port 80. Does traffic magically goes from 80 to 8080? Any ideas what is going on here?
You are forgetting to look at one part of that nginx config:
upstream nodejs {
server 127.0.0.1:8081;
keepalive 256;
}
That part is telling nginx to make a group of servers called nodejs as you can read about here.
8081 is the port that NodeJS is running on (if you use the sample application for instance).
You can verify this by looking at the Elastic Beanstalk logs:
-------------------------------------
/var/log/nodejs/nodejs.log
-------------------------------------
Server running at http://127.0.0.1:8081/
Then if we continue in the nginx.conf file we can see what you already posted:
server {
listen 8080;
location / {
proxy_pass http://nodejs;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
This tells nginx to use the proxy pass module to pass all from port 8080 to our upstream group nodejs which is running on port 8081. This means that port 8081 is just for accessing it locally but port 8080 is what let's outside entities talk to the nginx which then passes stuff onto nodejs.
Some of the reasoning for not exposing NodeJS directly can be found in this StackOverflow answer.
Port 8080 is used because it is the HTTP alternate port that is "commonly used for Web proxy and caching server, or for running a Web server as a non-root user."
That explains the ports. Now the issue of ELB and how things are talking to each other.
Since the security group is only allowing access on port 80, there is an iptables rule that is setup to forward port 80 to port 8080. This allows non-root to bind to port 8080 because lower port numbers require root privileges.
You can verify this by running the following:
[ec2-user#ip-xxx-xx-xx-x ~]$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- anywhere anywhere tcp dpt:http redir ports 8080
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- anywhere anywhere tcp dpt:http redir ports 8080
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
So in summary, when you load your CNAME, the load balancer is rerouting the traffic to a given instance on port 80, which is allowed through the security group, then iptables is forwarding that to port 8080, which is the port that nginx is using a proxy to pass the traffic to port 8081 which is the local port of NodeJS.
Here's a diagram:
incoming connections
-> :80 - Load Balancer
-> :80 - Security group
-> :80 -> :8080 - EC2 instance, iptables forward
-> :8080 -> :8081 - nginx, proxy pass
-> :8081 - nodejs, your app
Hopefully that helps.

amazon ec2 - Couldn't resolve host

I'm a nooob devops guy in town. I've a nodejs app running on amazon instance pointing to DNS's.
Route53 screenshot
Problem
when I try to reach the subdomain, I get
curl -i http://site.soothsayer.co/
curl: (7) Failed connect to site.soothsayer.co:80; Connection timed out
or
$ curl -i http://site.soothsayer.co/
HTTP/1.1 502 Bad Gateway
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8
Via: 1.1 Comverse 6.2.23
Content-Length: 287
<!DOCTYPE html PUbliC "-//W3C//Dtd XHTML basic 1.0//EN"
"http://www.w3.org/tr/1999/REC-html401-19991224/loose.dtd">
<html>
<head>
<title>Error</title>
</head>
<body leftmargin="0" topmargin="0" style="font-family:arial">
Error: Requested url content cannot be resolved.
</body>
</html>
Debugging
nslookup result is
Server: 127.0.1.1
Address: 127.0.1.1#53
Non-authoritative answer:
site.soothsayer.co canonical name = internal-p-site-us-east-1-1077850846.us-east-1.elb.amazonaws.com.
Name: internal-p-site-us-east-1-1077850846.us-east-1.elb.amazonaws.com
Address: 172.31.46.66
I can see proper html response from GET http://localhost:3000 or GET http://localhost:8085 (configured in nginx) within the amazon machine.
I'm using nginx/1.4.7 as HTTP and reverse proxy server (as per my understanding), conf file being
$ vi /etc/nginx/conf.d/site.soothsayer.co.conf
# the IP(s) on which your node server is running. I chose port 3000.
upstream site.soothsayer.co {
server 127.0.0.1:3000;
}
# the nginx server instance
server {
listen 0.0.0.0:8085;
server_name site-proxy.soothsayer.co site-proxy.soothsayer.co;
access_log /var/log/nginx/site-proxy.soothsayer.co.log;
# pass the request to the node.js server with the correct headers and much more can be added, see nginx config options
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://site.soothsayer.co/;
proxy_redirect off;
}
}
My /etc/resolv.conf is,
; generated by /sbin/dhclient-script
search ec2.internal
nameserver 172.31.0.2
iptables of amazon machine,
[root#ip-172-31-42-77 rof]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Security Group on amazon is default,
Custom TCP Rule TCP 8086 0.0.0.0/0
I don't see any logs at /var/log/nginx/error.log. This is what my little brain could debug. I want to fix this issue.
References
nginx, dns - domain.com resolves but subdomain.domain.com doesn't
AWS - elastic load balancer unable to handle the request
amazon ec2 instance unable to resolve host
Curl amazon EC2 instance
So, issue was with load balancer being internal. Made it internet-facing. Everything looks Majestic now.
References
Internet-facing and Internal Load Balancers
not sure, but your nginx conf with listen 0.0.0.0:8085 may block any remote access.
Try :
listen 8085;