I have my app running on EKS which is using istio-ingressgateway service for load balancer and Knative serving I have added ACM to my ELB, but after patching the service with
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:xx-xxxx-1:1234567890:certificate/xxxxxx-xxx-dddd-xxxx-xxxxxxxx"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
my domain is not opening on HTTPS but works fine on HTTP giving this error on HTTPS
< HTTP/1.1 408 REQUEST_TIMEOUT
HTTP/1.1 408 REQUEST_TIMEOUT
< Content-Length:0
Content-Length:0
< Connection: Close
Connection: Close
Hope you your load balancer forward the traffic from 443 to the backend target port 3190 in case of Istio. Check your Istio gateway file wether you have 443 port mapped with the targets.
Related
I have installed nginx ingress controller of type NLB inside EKS cluster and it is of type internal.
The ingress controller created a network load balancer, with listeners 80 and 443,
with port 443 we can't attach an ssl cert for nlb type, only when I use listener type tls it is able to allow us to add ssl cert from AWS ACM.
Now the issue is, I am trying to expose a frontend application through this NLB nginx ingress controller,
when the NLB lister port is 443, it is able to access the application but complains with ssl cert (fake Kubernetes cert), when I change the listener from 443 to tls in NLB, it throws error "400 "The plain HTTP request was sent to HTTPS port" error"
Like many solutions out there mentioning changing the targetPort from https: https to https: http , I tried but with that too same error "The page isn't working,ERR_TOOMANY_REQUESTS"
Could anyone help me how to resolve this issue?
Any ideas or suggestions would be highly appreciated
To resolve the issue with the SSL certificate and the "400 "The plain HTTP request was sent to HTTPS port" error", you may need to modify your ingress configuration to specify that the ingress should listen for HTTPS traffic on port 443. This can be done by adding the following annotations to your ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/secure-backends: "true"
name: example
namespace: example
spec:
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: example
port:
name: https
tls:
- hosts:
- example.com
secretName: example-tls
In the example above, nginx.ingress.kubernetes.io/ssl-redirect tells the ingress to redirect HTTP traffic to HTTPS. nginx.ingress.kubernetes.io/secure-backends tells the ingress to encrypt the traffic between the ingress and the backend services. `secret
I'm using Nginx ingress controller in kubernetes cluster and its host address (load balancer DNS name) is returning "Could not resolve host: ..".
In AWS, I don't even have load balancer with this DNS name.
However, when I run kubectl logs on ingress controller, it's still receiving traffic normally.
How is it possible for ingress controller to have a host that can't be found and still receives traffic?
Nginx Ingress Controller service
-> kubectl -n ingress-nginx describe svc ingress-nginx-controller
...
LoadBalancer Ingress: xxx.elb.ap-northeast-2.amazonaws.com #deprecated
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 31610/TCP
Endpoints: 10.0.51.53:80
Port: https 443/TCP
TargetPort: http/TCP
NodePort: https 32544/TCP
Endpoints: 10.0.51.53:80
Session Affinity: None
External Traffic Policy: Cluster
Events:
Warning SyncLoadBalancerFailed 32m (x6919 over 24d) service-controller (combined from similar events): Error syncing load balancer: failed to ensure load balancer: error creating load balancer listener: "TargetGroupAssociationLimit:
The following target groups cannot be associated with more than one load balancer: arn:aws:elasticloadbalancing:ap-northeast-2:4xxx:targetgroup/k8s-ingressn-ingressn-5f8ebc7e16/25aa0ef278298505\n\tstatus
code: 400, request id: d1767330-f7d1-4c4f-bcf1-4f1e4af8ab9f"
Normal
EnsuringLoadBalancer 2m40s (x6934 over 24d) service-controller Ensuring load balancer
-> curl xxx.elb.ap-northeast-2.amazonaws.com
curl: (6) Could not resolve host: xxx.elb.ap-northeast-2.amazonaws.com
Ingress using deprecated DNS
-> kubectl describe ingress slack-api-ingress
Name: slack-api-ingress
Namespace: default
Address: xxx.elb.ap-northeast-2.amazonaws.com #deprecated
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
...
I have another NLB in AWS which was initially provisioned by ingress controller and I can still access my services with this lb's DNS name.
I'm assuming that ingress controller tried to update and create a new load balancer, but still directs traffic to previous lb after failing to create new lb. If this is the case, I want ingress-controller to use previous lb's DNS.
I'm new to kubernetes and probably don't know what's going on so any advice is appreciated.
I'm using EKS and latest Istio installed via Helm. I'm trying to implement TLS based on a wildcard cert we have for our domain in AWS certificate manager. I'm running into a problem where the connection between the client and the NLB works, with TLS being terminated there, but the NLB can't talk to the istio LB over the secure port. In the AWS console I can rewrite the forwarding rules to forward traffic from port 443 to the standard istio http target, but I can't find a way to do this via code. I'm trying to avoid all click-ops. Here is my Helm overrides for the gateway:
gateways:
istio-ingressgateway:
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:XXXXXXXXXXXXXXXXXX:certificate/XXXXXXXXXXXXXXXXXXXX"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
So what I'm expecting to occur here is:
Client:443 --> NLB:443 --> Istio Gateway:80
but what I end up with is
Client:443 --> NLB:443 --> Istio Gateway:443
Does anyone have any thoughts on how to get this to work via code? Alternately if someone can point me to a resource to get tls communication between the NLB and Istio working I'm good with that too.
Probably, what is happening is that if you terminate TLS on the load balancer it won't carry SNI to the target group. I had the exact same issue and I ended up solving it by setting the host as '*' on the ingress Gateway and then specifying the hosts on the different VirtualServices (as recommended here and also on istio's official docs).
Your service annotation already correct what is mising is to change istio gateway port 443 to HTTP
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: http-gateway-external
namespace: istio-ingress
spec:
selector:
istio: gateway-external
servers:
- hosts:
- '*'
port:
name: http
number: 80
protocol: HTTP
- hosts:
- '*'
port:
name: https
number: 443
protocol: HTTP # Change from HTTPS to HTTP
I have a cluster created using eksctl and also valid certificates created under ACM, I have used DNS method to verify the domain ownership and its succesfully completed.
below are the details i see when executing kubectl get ing
NAME HOSTS ADDRESS PORTS AGE
eks-learning-ingress my-host.com b29c58-production-91306872.us-east-1.elb.amazonaws.com 80 18h
when i access the application using https://b29c58-production-91306872.us-east-1.elb.amazonaws.com, i see it load the application with a security warning because that not the domain name with which the certifcates are created. When i try to execute https://my-host.com i am getting a timeout.
I have 2 questions
1) I am using CNAMES to point my domain to AWS ELB, the values i added for CNAME are
name: my-host.com, points to: b29c58-production-91306872.us-east-1.elb.amazonaws.com. Is this correct?
2) below is my ingress resource defination, may be i am missing something as requests are not coming in to the application
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: eks-learning-ingress
namespace: production
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/certificate-arn: arn:dseast-1:255982529496:sda7-a148-2d878ef678df
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}, {"HTTP": 8080}, {"HTTPS": 8443}]'
labels:
app: eks-learning-ingress
spec:
rules:
- host: my-host.com
http:
paths:
- path: /*
backend:
serviceName: eks-learning-service
servicePort: 80
Any help would be really great. Thanks.
My go-to solution is using an A-record in Route 53. Instead of adding an IP, you select the "alias" option and select your load balancer. Amazon will handle the rest.
I think you have a port mismatch. https:// will use port 443, not port 80, but your ingress appears to be accepting requests on port 80 rather than 443.
If 443 was configured I'd expect to see it listed under ports as 80, 443
Can you verify with telnet or nc or curl -vvvv that your ingress is actually accepting requests on port 443? If it is, check the response body reported by curl - it should give you some indication as to why the request is not propagating downwards to your service.
We use nginx-ingress so unfortunately I can't look at local ingress config and compare it to yours.
We are running rails application with unicorn and websocket.
We are using AWS ELB as ingress
SSL terminates on ELB and forwards traffic to application.
Nginx ingress routes traffic to web app running unicorn/puma on port 8080.
App works but our websocket responds with 200 instead of 101. We have enabled CORS and used required annotations in ingress.
This are annotations used for the ingress controller service
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
service.beta.kubernetes.io/aws-load-balancer-ssl-cert::arn:aws:iam::xxx:server-certificate/staging
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
When we use aws loadbalancer protocol as tcp and load balancer ports as 443 it fails on infinite redirect loop.
Following are the annotations used in the ingress:
nginx.ingress.kubernetes.io/service-upstream: true
nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"
nginx.ingress.kubernetes.io/cors-allow-headers: "DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type"
nginx.ingress.kubernetes.io/cors-allow-origin: "*"
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
ingress.kubernetes.io/force-ssl-redirect: "true"
Our sample nginx configuration we used earlier without ingress is here
How to get websockets working with nginx ingress controller with AWS ELB ?
Is it possible to try without CORS?
Part of the handshake is the client must send at least these headers:
Sec-WebSocket-Key
Sec-WebSocket-Version
And maybe something else. Look at https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#The_WebSocket_Handshake