My application consists of play web application deployed using GKE. The application was running fine (using Deployment and Loadbalancer service) and then I decided to use Ingress. I made the following changes which has made the application unreachable. I get 502 error when I try to connect with the application using ingress IP.
The application is of kind Deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
spec:
replicas: 2
selector:
matchLabels:
app: webapp
It has a service associated with it
apiVersion: v1
kind: Service
metadata:
name: webapp-service
spec:
selector:
app: webapp
ports:
- protocol: TCP
port: 9000 #this service is reachable at this port
targetPort: 9000 #this service will forward the request to correspoding nodes of the service at this port
#type: LoadBalancer
type: NodePort
Then I applied the following file to create ingress
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: webapp-https-loadbalancer-ingress
annotations:
kubernetes.io/ingress.class: "gce"
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: webapp-service
servicePort: 9000
I can see that there is an IP address (which is also reachable from outside) when I run
kubectl describe ingress webapp-https-loadbalancer-ingress
Name: webapp-https-loadbalancer-ingress
Namespace: default
Address: 3x.yyy.zzz.pq
Default backend: default-http-backend:80 (10.88.0.5:8080)
Rules:
Host Path Backends
---- ---- --------
*
... Type Reason Age From Message
---- ------ ---- ---- -------
Normal ADD 10m loadbalancer-controller default/webapp-https-loadbalancer-ingress
Normal CREATE 9m10s loadbalancer-controller ip: 3x.yyy.zzz.pq
But I am not able to reach the application using https://3x.yyy.zzz.pq. I haven't yet associated the domain with the IP. I tried to connect using curl and got error- 502 bad gateway error
curl -v 3x.xxx.xxx.xxx
* Expire in 0 ms for 6 (transfer 0x55d4c5258f90)
* Trying 3x.xxx.xxx.xxx...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55d4c5258f90)
* Connected to 3x.xxx.xxx.xxx (3x.xxx.xxx.xxx) port 80 (#0)
> GET / HTTP/1.1
> Host: 3x.xxx.xxx.xxx
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 502 Bad Gateway
< Content-Type: text/html; charset=UTF-8
< Referrer-Policy: no-referrer
< Content-Length: 332
< Date: Tue, 22 Dec 2020 22:27:23 GMT
<
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>
* Connection #0 to host 3x.xxx.xxx.xxx left intact
The issue was that the load balancer's IP was not in the allowed-hosts list in the server. As temporary fix, I used wild card in server configuration to allow traffic from all hosts. I am still figuring out how to restrict it to load balancer's internal IPs
Related
I am trying to setup mTLS for outgoing connections, but instead of originating the TLS traffic from the egress gateway, I’m trying to do it from the sidecar proxy itself. We want to originate the TLS connection from proxy and not the egress gateway.
I took care of mounting the client certs in my sidecar proxy container and verified that the client certs are available in the expected path. My API resources look something like below
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: external-host-mtls
spec:
hosts:
- external-host-example.com
location: MESH_EXTERNAL
ports:
- number: 443
name: https
protocol: HTTPS
resolution: DNS
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: external-mtls
spec:
hosts:
- external-host-example.com
tls:
- match:
- port: 443
sniHosts:
- external-host-example.com
route:
- destination:
host: external-host-example.com
port:
number: 443
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: external-mtls
spec:
host: external-host-example.com
trafficPolicy:
tls:
mode: MUTUAL
clientCertificate: /etc/client-certs/client.pem
privateKey: /etc/client-certs/client.key
caCertificates: /etc/client-certs/ca.pem
when I try to curl to external-host-example.com, I am hoping that Istio will add the client certs to the connection.
I’m not sure if that’s happening, because I’m running into errors.
curl -H "Host: external-host-example.com" --tlsv1.2 -v https://external-host-example.com
* About to connect() to external-host-example.com port 443 (#0)
* Trying x.x.x.x...
* Connected to external-host-example.com (x.x.x.x) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file
Looking at the debug logs, I see this
|2020-11-17T15:53:58.226367Z|debug|envoy filter|[external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:148] tls:onServerName(), requestedServerName: external-host-example.com|
|2020-11-17T15:53:58.226443Z|debug|envoy filter|[external/envoy/source/common/tcp_proxy/tcp_proxy.cc:251] [C161] new tcp proxy session|
|2020-11-17T15:53:58.226480Z|debug|envoy filter|[external/envoy/source/common/tcp_proxy/tcp_proxy.cc:395] [C161] Creating connection to cluster outbound|443||external-host-example.com|
|2020-11-17T15:53:58.226509Z|debug|envoy pool|[external/envoy/source/common/tcp/conn_pool.cc:83] creating a new connection|
|2020-11-17T15:53:58.226550Z|debug|envoy pool|[external/envoy/source/common/tcp/conn_pool.cc:364] [C162] connecting|
|2020-11-17T15:53:58.226557Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:727] [C162] connecting to x.x.x.x:443|
|2020-11-17T15:53:58.226641Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:736] [C162] connection in progress|
|2020-11-17T15:53:58.226656Z|debug|envoy pool|[external/envoy/source/common/tcp/conn_pool.cc:109] queueing request due to no available connections|
|2020-11-17T15:53:58.226662Z|debug|envoy conn_handler|[external/envoy/source/server/connection_handler_impl.cc:411] [C161] new connection|
|2020-11-17T15:53:58.252446Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:592] [C162] connected|
|2020-11-17T15:53:58.252555Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C162] handshake expecting read|
|2020-11-17T15:53:58.277388Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C162] handshake expecting read|
|2020-11-17T15:53:58.277417Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C162] handshake expecting read|
|2020-11-17T15:53:58.277595Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:176] [C162] handshake complete|
|2020-11-17T15:53:58.277633Z|debug|envoy pool|[external/envoy/source/common/tcp/conn_pool.cc:285] [C162] assigning connection|
|2020-11-17T15:53:58.277661Z|debug|envoy filter|[external/envoy/source/common/tcp_proxy/tcp_proxy.cc:624] TCP:onUpstreamEvent(), requestedServerName:external-host-example.com|
|2020-11-17T15:53:58.303804Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:226] [C162]|
|2020-11-17T15:53:58.303830Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:558] [C162] remote close|
|2020-11-17T15:53:58.303834Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:200] [C162] closing socket: 0|
|2020-11-17T15:53:58.303853Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:298] [C162] SSL shutdown: rc=-1|
|2020-11-17T15:53:58.303855Z|debug|envoy connection|[external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:226] [C162]|
|2020-11-17T15:53:58.303880Z|debug|envoy pool|[external/envoy/source/common/tcp/conn_pool.cc:124] [C162] client disconnected|
|2020-11-17T15:53:58.303894Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:109] [C161] closing data_to_write=0 type=0|
|2020-11-17T15:53:58.303900Z|debug|envoy connection|[external/envoy/source/common/network/connection_impl.cc:200] [C161] closing socket: 1|
|2020-11-17T15:53:58.303985Z|debug|envoy conn_handler|[external/envoy/source/server/connection_handler_impl.cc:111] [C161] adding to cleanup list|
Any idea what am I doing wrong? How do I debug this further?
I am trying to host a django website on Azure kubernetes service behide nginx-ingress, and I would like my django web show under a path.
e.g. when access the default admin site, I would like to access it at http://example.com/django/admin instead of http://example.com/admin
I tried the configure below, when I access http://example.com/django/admin it will forward me to http://example.com/admin and show me 404 error from default ingress backend, as I set django debug to ture I assume this mean ingress did not send my request to my django service
# path example
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: django-ingress
labels:
app: django
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- backend:
serviceName: django-service
servicePort: 80
path: /django(/|$)(.*)
so I try to curl -I -k http://example.com/django/admin, and it show something like below
HTTP/1.1 301 Moved Permanently
Server: openresty/1.15.8.2
Date: Wed, 06 Nov 2019 04:14:14 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Location: /admin/
The same thing happen to any valid page in the site, if I curl -I -k http://example.com/django/any_valid_page it show below:
HTTP/1.1 301 Moved Permanently
Server: openresty/1.15.8.2
Date: Wed, 06 Nov 2019 04:14:14 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Location: /any_valid_page/
I wonder it is caused by I am doing the test with the default django development web server? (i.e. python manage.py runserver)
If I try to host it at root like below, everything is fine...
# root example
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: django-ingress
labels:
app: django
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- backend:
serviceName: django-service
servicePort: 80
path: /
Trying adding this
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: django-ingress
labels:
app: django
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
ingress.kubernetes.io/rewrite-target: /django
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- backend:
serviceName: django-service
servicePort: 80
path: /django
Starting in Version 0.22.0, ingress definitions using the annotation
nginx.ingress.kubernetes.io/rewrite-target are not backwards
compatible with previous versions. In Version 0.22.0 and beyond, any
substrings within the request URI that need to be passed to the
rewritten path must explicitly be defined in a capture group. So make
sure you have right version.
When using SSL offloading outside of cluster it may be useful to enforce a redirect to HTTPS even when there is no TLS certificate available. This can be achieved by using the nginx.ingress.kubernetes.io/force-ssl-redirect: "true" annotation in the particular resource.
I think your Ingress configuration file should look like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: django-ingress
labels:
app: django
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: example.com
http:
paths:
- path: /django(/|$)(.*)
backend:
serviceName: django-service
servicePort: 80
If you get 404 error, there is possible solution:
Please change https to http in the curl command?:
curl --resolve your-host:80:xx.xxx.xx.xxx http://my-host:80
To get the IP from kubectl get ing command, it is necessary to
enable the reporting Ingress status feature. Take a look on: reporting-ingress-status.
There is the default server in the Ingress controller. It returns
the Not Found page with the 404 status code for all requests for
domains for which there are no Ingress rules defined. Those requests
are not shown in the access log.
Since you're getting a 404, this means that the host header of your
requests doesn't match with the host field in the Ingress resource.
To set the host header in curl, please see previous curl
commands. Optionally, you can also do:
curl http://<ip> -H "host: example.com"
Please take a look on ngnix-ingress, server-side-https-enforcement-nginx.
This is a problem from Django's side. Whenever the admin is not logged in, the /django/admin results in a redirect to /admin/. In this case, if you just replace /django/admin/ with /django/admin/ in the browser URL field it will work and open django admin login.
So basically Django's built-in redirect conflicts with the Ingress's rewrite module.
Trying to curl the service deployed in k8s cluster from the master node
curl: (7) Failed to connect to localhost port 31796: Connection
refused
For kubernetes cluster, when I check my iptables on master I get the following .
Chain KUBE-SERVICES (1 references)
target prot opt source destination
REJECT tcp -- anywhere 10.100.94.202 /*
default/some-service: has no endpoints */ tcp dpt:9015 reject-with
icmp-port-unreachable
REJECT tcp -- anywhere 10.103.64.79 /*
default/some-service: has no endpoints */ tcp dpt:9000 reject-with
icmp-port-unreachable
REJECT tcp -- anywhere 10.107.111.252 /*
default/some-service: has no endpoints */ tcp dpt:9015 reject-with
icmp-port-unreachable
if I flush my iptables with
iptables -F
and then curl
curl -v localhost:31796
I get the following
* Rebuilt URL to: localhost:31796/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 31796 (#0)
> GET / HTTP/1.1
> Host: localhost:31796
> User-Agent: curl/7.58.0
> Accept: */*
but soon after it results in
* Rebuilt URL to: localhost:31796/
* Trying 127.0.0.1...
* TCP_NODELAY set
* connect to 127.0.0.1 port 31796 failed: Connection refused
* Failed to connect to localhost port 31796: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 31796: Connection
refused
I'm using the nodePort concept in my service
Details
kubectl get node
NAME STATUS ROLES AGE VERSION
ip-Master-IP Ready master 26h v1.12.7
ip-Node1-ip Ready <none> 26h v1.12.7
ip-Node2-ip Ready <none> 23h v1.12.7
Kubectl get pods
NAME READY STATUS RESTARTS AGE
config-service-7dc8fc4ff-5kk88 1/1 Running 0 5h49m
kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
AGE SELECTOR
cadmin-server NodePort 10.109.55.255 <none>
9015:31796/TCP 22h app=config-service
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
26h <none>
Kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
endpoint.yml
apiVersion: v1
kind: Endpoints
metadata:
name: xyz
subsets:
- addresses:
- ip: node1_ip
- ip: node2_ip
ports:
- port: 31796
- name: xyz
service.yml
apiVersion: v1
kind: Service
metadata:
name: xyz
namespace: default
annotations:
alb.ingress.kubernetes.io/healthcheck-path: /xyz
labels:
app: xyz
spec:
type: NodePort
ports:
- nodePort: 31796
port: 8001
targetPort: 8001
protocol: TCP
selector:
app: xyz
deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: xyz
name: xyz
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: xyz
template:
metadata:
labels:
app: xyz
spec:
containers:
- name: xyz
image: abc
ports:
- containerPort: 8001
imagePullPolicy: Always
resources:
requests:
cpu: 200m
volumeMounts:
- mountPath: /app/
name: config-volume
restartPolicy: Always
imagePullSecrets:
- name: awslogin
volumes:
- configMap:
name: xyz
name: config-volume
You can run the following command to check endpoints.
kubectl get endpoints
If endpoint is not showing up for the service. Please check the yml files that you used for creating the loadbalancer and the deployment. Make sure the labels match.
As many have pointed out in their comments the Firewall Rule "no endpoints" is inserted by the kubelet service and indicates a broken Service Application Definition or Setup.
# iptables-save
# Generated by iptables-save v1.4.21 on Wed Feb 24 10:10:23 2021
*filter
# [...]
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "default/web-service:http has no endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 30081 -j REJECT --reject-with icmp-port-unreachable
# [...]
As you have noticed as well the service kubelet constantly monitors the Firewall Rules and inserts or deletes rules dynamically according to the Kubernetes Pod or Service definitions.
# kubectl get service --namespace=default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 198d
web-service NodePort 10.111.188.199 <none> 8201:30081/TCP 194d
# kubectl get pods --namespace=default
No resources found in default namespace.
In this example case a Service is defined but the Pod associated with the Service does not exist.
Still the kube-proxy process listens on the port 30081:
# netstat -lpn | grep -i kube
[...]
tcp 0 0 0.0.0.0:30081 0.0.0.0:* LISTEN 21542/kube-proxy
[...]
So the kubelet service inserts a firewall rule to prevent the traffic for the broken service.
Also the kubelet service will delete this rule as soon as you delete the Service definition
# kubectl delete service web-service --namespace=default
service "web-service" deleted
# iptables-save | grep -i "no endpoints" | wc -l
0
As a Side Node:
This rule is also inserted for Kubernetes Definitions that the kubelet Service doesn't like.
As an example your service can have the name "log-service" but can't have the name "web-log".
In the latter case the kubelet Service didn't give a warning but inserted this blocking rule
I have a simple Flask app. It worked fine when I connected to it via port-forwarding to send the HTTP Post request directly to the Service.
from flask import Flask, request
import redis
from rq import Queue
from worker import job_worker
UPLOAD_FOLDER = './uploads/'
app = Flask(__name__)
r = redis.Redis()
q = Queue(connection = r)
#app.route('/', methods=['POST'])
def upload():
scale = int(request.form['scale'])
q.enqueue(job_worker, scale)
return ""
if __name__ == "__main__":
app.run()
I also have a simple index.html file in an nginx container which is served at port 80. It does an ajax POST request to "/upload". Which if you look at the ingress controller, should convert that to a port 5000 request and strip away the "upload"
The flask app gets served at port 5000
Here is the ingress controller:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: emoji-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /upload
backend:
serviceName: emoji-backend
servicePort: 5000
- path: /
backend:
serviceName: emoji-frontend
servicePort: 80
And for completeness, the emoji-backend service:
apiVersion: v1
kind: Service
metadata:
name: emoji-backend
labels:
app: emoji-backend
tier: backend
spec:
type: LoadBalancer
ports:
- port: 5000
selector:
app: emoji-backend
tier: backend
I get a 502 bad gateway without really any indication except the ingress log does say this:
2019/09/29 21:41:04 [error] 2021#2021: *78651 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.64.1, server: _,
request: "POST /upload HTTP/2.0", upstream: "http://172.17.0.4:5000/", host: "192.168.64.5", referrer: "https://192.168.64.5/"
"http://172.17.0.4:5000/" is the correct endpoint and port for the emoji-backend service.
Adding the following line fixed it:
app.run(debug=True,host='0.0.0.0',port=5000)
However, it took me a while to figure that out because at first when I tried it my docker image was not updating when I re-deployed.
I try to configure Google Cloud CDN to my container engine project.
Following the documentation It has either a Content-Length header or a Transfer-Encoding header in order to be cached.
My backend use gzip compression so I have Transfer-Encoding: chunked
The problem is it seems the ingress load balancer remove the Transfer-encoding header so I can't have a "cache hit"
I used "kubectl port-forward" to connect direclty to an instance backend and I have the Transfer-encoding header.
But when I connect to the external IP, the header has disappear.
Here my ingress configuration
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway-preprod3-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: gateway-preprod2-static-ip
spec:
tls:
- secretName: gateway-preprod-secret-2018-with-ca-7
backend:
serviceName: gateway-preprod
servicePort: 80
Here my deployment configuration
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gateway-preprod
spec:
replicas: 1
strategy:
type: RollingUpdate
minReadySeconds: 50
template:
metadata:
labels:
app: gateway-preprod
spec:
containers:
- name: gateway-preprod
image: eu.gcr.io/writecontrol-1055/gateway:v305
env:
- name: writecontrolEnv
value: preprod
ports:
- containerPort: 80
In the opposite, for certain ressources not GZIP compressed, the Content-length header is given and I have a successful "cache hit"
The kubernetes version is 1.7.12-gke.1
Here an URL to test it : https://preprod-writecontrol.ovh
I am not sure why your response has Transfer-Encoding header. This header is not meant to be at the source (your application) and it is usually added by other hops, such as proxies like Cloud HTTP Load Balancer (=Ingress).
More info on Transfer-Encoding here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
I was able to get CDN working with a GZIP response myself using the Content-Encoding header instead:
First read the caching requirements at https://cloud.google.com/cdn/docs/caching. You need a Content-Length and/or Cache-Control: public header. So code these headers into your application server.
After enabling CDN on the load balancer created by Ingress, make two requests, and if you see Age header on the second request (as I did below), it means your request is now being served from CDN.
curl -vH Accept-Encoding:gzip 35.186.195.233
> [...]
>
< HTTP/1.1 200 OK
< Content-Encoding: gzip
< Date: Tue, 27 Mar 2018 19:38:20 GMT
< Content-Length: 87
< Content-Type: application/x-gzip
< Via: 1.1 google
< Cache-Control: max-age=600,public
��H����Q(�/�IQ�
* Connection #0 to host 35.186.195.233 left intact
K-*��ϳR0�3�3���/.�K�M�R�)+OM�55575��L�4ѭ�N+L���K��4A
And the second request (note the Age header):
$ curl -vH Accept-Encoding:gzip 35.186.195.233
[...]
>
< HTTP/1.1 200 OK
< Cache-Control: max-age=600,public
< Content-Encoding: gzip
< Date: Tue, 27 Mar 2018 19:42:01 GMT
< Content-Length: 87
< Content-Type: application/x-gzip
< Via: 1.1 google
< Age: 314
<
��H����Q(�/�IQ�
* Connection #0 to host 35.186.195.233 left intact
K-*��ϳR0�3�3���/.�K�M�R�)+OM�55575��L�4ѭ�N+L���K��4A