I have created one load balancer service using the following configuration:-
apiVersion: v1
kind: Service
metadata:
name: dealer-service-elb
namespace: staging
labels:
service: dealer-service-elb
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
spec:
type: LoadBalancer
selector:
service: dealer-service
ports:
- name: 'http'
port: 80
targetPort: 3000
protocol: 'TCP'
and my aws alb looks like this:-
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dealer-service-ingress
labels:
app: dealer-service
namespace: staging
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/healthcheck-path: /dealers-service/v1/health
alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: dealer-service
servicePort: 80
But when I checked my ingress using kubectl command:-
I did not find my ingress url:-
NAME HOSTS ADDRESS PORTS AGE
dealer-service-ingress * 80 34m
Related
I have an EKS kubernetes cluster with AWS Load Balancer Controller and Argo CD installed. I'm creating an Application Load Balancer based on Argo CD documentation here.
Basically, I'm creating a NodePort service that receives traffic from the load balancer, and an ingress that will create the load balancer (using AWS Load Balancer Controller).
The ingress code is this one:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/backend-protocol: HTTPS
alb.ingress.kubernetes.io/backend-protocol-version: HTTP2
# Use this annotation (which must match a service name) to route traffic to HTTP2 backends.
alb.ingress.kubernetes.io/conditions.argogrpc: |
[{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "Content-Type", "values":["application/grpc"]}}]
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
# ALB annotations
kubernetes.io/ingress.class: 'alb'
alb.ingress.kubernetes.io/scheme: 'internet-facing'
alb.ingress.kubernetes.io/target-type: 'instance'
alb.ingress.kubernetes.io/load-balancer-name: 'test-argocd'
alb.ingress.kubernetes.io/certificate-arn: 'arn:aws:acm:us-east-1:1234567:certificate/longcertcode'
alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true
# Health Check
alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
name: argocd
namespace: argocd
spec:
rules:
- host: argocd.argoproj.io
http:
paths:
- path: /
backend:
service:
name: argogrpc
port:
number: 443
pathType: ImplementationSpecific
tls:
- hosts:
- argocd.argoproj.io
defaultBackend:
service:
name: argogrpc
port:
number: 443
And that creates a Load Balancer as expected.
I'm creating the service with this:
apiVersion: v1
kind: Service
metadata:
annotations:
alb.ingress.kubernetes.io/backend-protocol-version: HTTP2
labels:
app: argogrpc
name: argogrpc
namespace: argocd
spec:
ports:
- name: "443"
port: 443
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: argocd-server
sessionAffinity: None
type: NodePort
The issue here is that the health check is failing on the Target Group:
If I change the backend protocol version to GRPC:
apiVersion: v1
kind: Service
metadata:
annotations:
alb.ingress.kubernetes.io/backend-protocol-version: GRPC
labels:
app: argogrpc
name: argogrpc
namespace: argocd
spec:
ports:
- name: "443"
port: 443
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: argocd-server
sessionAffinity: None
type: NodePort
Then the health check is passed, but I get an 464 error on Chrome:
This is what AWS documentation says about this error, but it doesn't help to clarify why I'm getting it:
So the question is, how do I create an application load balancer for my Argo CD using AWS load balancer controller that works? According to the documentation, it should work in both cases.
Ingress is not routing traffic to backend service port. Connection times out.
Any idea how to troubleshoot? There are no errors in the ingress controller logs.
curl -v -L https://www.example.com
* Trying 172.20.xxx.xx:443...
I've checked the Loadbalancer and its health is fine. I'm not sure why its not routing traffic to backend port and what else I could check to figure out what the issue is.
Here is my aws ingress controller.yaml config :
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:xxx"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.3.0
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: http
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: LoadBalancer
And here the service config service.yaml:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2022-08-06T12:16:17Z"
labels:
app: manageiq
name: httpd
namespace: default
spec:
clusterIP: 100.69.xxx.xx
clusterIPs:
- 100.69.xxx.xx
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
name: httpd
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
Ingress.yaml:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
creationTimestamp: "2022-08-06T12:47:32Z"
generation: 2
labels:
app: manageiq
name: httpd
namespace: default
spec:
rules:
- host: www.example.com
http:
paths:
- backend:
service:
name: httpd
port:
number: 8080
path: /
pathType: ImplementationSpecific
status:
loadBalancer:
ingress:
- hostname: adc1d30f0db264d7ea54aed4dcdc12ec-atest.elb.ap-south-1.amazonaws.com
Figured it out. Loadbalancer was set to internal. I was following a documentation.
service.beta.kubernetes.io/aws-load-balancer-internal: "false"
I had to delete services and recreate so a new load balancer gets created.
When we launch the EKS Cluster using the below manifest, it is creating ALB. We have a default ALB that we are using, let's call it EKS-ALB. The Hosted zone is routing traffic to this EKS-ALB. We gave tag ingress.k8s.aws/resource:LoadBalancer, ingress.k8s.aws/stack:test-alb, elbv2.k8s.aws/cluster: EKS. But when we delete the manifest, it is deleting the default ALB and we need to reconfigure hosted zone again with New ALB which will get created in next deployment. Is there any way to block Ingress-controller not deleting ALB, but only deleting the listeners and Target Group?
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test-nginx-rule
namespace: test
annotations:
alb.ingress.kubernetes.io/group.name: test-alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
alb.ingress.kubernetes.io/ssl-redirect: '443'
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
alb.ingress.kubernetes.io/healthcheck-path: /index.html
alb.ingress.kubernetes.io/success-codes: 200-399
alb.ingress.kubernetes.io/security-groups: eks-test-alb-sg
spec:
ingressClassName: alb
rules:
- host: test.eks.abc.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: test-svc
port:
number: 5005
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-dep
namespace: test
labels:
app: test
spec:
replicas: 1
restartPolicy:
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- name: test
image: Imagepath
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5005
resources:
requests:
memory: "256Mi"
cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
name: test-svc
namespace: test
labels:
app: test
spec:
type: NodePort
ports:
- port: 5005
targetPort: 80
protocol: TCP
selector:
app: test
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: test-scaler
namespace: test
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: test-dep
minReplicas: 1
maxReplicas: 5
targetCPUUtilizationPercentage: 60
---
In order to achieve the existing ALB not being deleted with group.name annotation enabled, we need to meet following conditions:
ALB should be tagged with below 3 tags:
alb.ingress.kubernetes.io/group.name: test-alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: instance
Create a dummy ingress with the same group name with the below manifest.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test-nginx-rule
namespace: test
annotations:
alb.ingress.kubernetes.io/group.name: test-alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
alb.ingress.kubernetes.io/ssl-redirect: '443'
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
alb.ingress.kubernetes.io/healthcheck-path: /index.html
alb.ingress.kubernetes.io/success-codes: 200-399
alb.ingress.kubernetes.io/security-groups: eks-test-alb-sg
spec:
ingressClassName: alb
rules:
- host: dummy.eks.abc.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: test-svc
port:
number: 5005
After deploying the above manifest, an ingress will be created using the same ALB and listener will have rule of if host is dummy.eks.abc.com, it will return 443. It's create and forget type of manifest, so after creating this ingress, even after we delete all the running deployment services (except the dummy manifest file above), the ALB will remain.
I want to secure my web service running on Kubernetes (EKS). It is running on port 80 .I want to run this on port 443.
When I apply the YAML file (for service and ingress), on AWS console I still have it listening on port 80 (and not on 443):
This is my YAML file:
How can I let it works? Thanks for you time!
#SERVICE LOGGER
apiVersion: v1
kind: Service
metadata:
name: load-balancer-api-logger
namespace: servicename-core-ns
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:786543355018:certificate/acdff29d4-7a32-42f1-8f11-1d4f495a5c77
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/force-ssl-redirect: "true"
spec:
selector:
app: api-logger
type: NodePort
ports:
- protocol: TCP
port: 443
targetPort: 5000
selector:
app.kubernetes.io/name: api-logger
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-articor
namespace: servicename-core-ns
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/healthcheck-path: "/healthcheckep"
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
alb.ingress.kubernetes.io/ssl-redirect: '443'
spec:
rules:
- host: logger.domainname.com
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: load-balancer-api-logger
port:
number: 80
Please consider that if I try to manually set the ALB to work with HTTPS it works fine. What I'm trying to achive here is to configure it via YAML file.
You should configure all settings in Ingress object. The following spec also don't repeat the default value set by the controller:
apiVersion: v1
kind: Service
metadata:
name: load-balancer-api-logger
namespace: servicename-core-ns
spec:
selector:
app: api-logger
type: NodePort
ports:
- protocol: TCP
port: 443
targetPort: 5000
selector:
app.kubernetes.io/name: api-logger
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-articor
namespace: servicename-core-ns
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/healthcheck-path: "/healthcheckep"
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-central-1:786543355018:certificate/acdff29d4-7a32-42f1-8f11-1d4f495a5c77
spec:
rules:
- host: logger.domainname.com
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: load-balancer-api-logger
port:
number: 443
I have created two ingresses, one for Grafana and one for my App. When the external dns write them into the route53 hosted zone as a A record, only one of them (the Myapp dns) get the (E)LB alias (dns), though the second A record get the internal IP as an ip address in the route53 A record.
the big question:
is there a way to set them all to the same alias/or at list to the same elb?
why it doesn't success doing so by default?
using:
terraform:
helm_release:
nginx-controller-bitnami
external-dns-bitnami
prometheus community
grafana-ingress:
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
meta.helm.sh/release-name: kube-prometheus-stack
meta.helm.sh/release-namespace: prometheus
generation: 1
labels:
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: grafana
app.kubernetes.io/version: 8.3.3
helm.sh/chart: grafana-6.20.5
name: kube-prometheus-stack-grafana
namespace: prometheus
resourceVersion: "2419"
spec:
rules:
- host: grafana.dns.io
http:
paths:
- backend:
service:
name: kube-prometheus-stack-grafana
port:
number: 80
path: /
pathType: Prefix
status:
loadBalancer:
ingress:
- ip: 10.0.1.19
kind: List
metadata:
resourceVersion: ""
selfLink: ""
my-app-ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
annotations:
ingress.kubernetes.io/ssl-redirect: /
nginx.ingress.kubernetes.io/rewrite-target: /$1
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: "some.website.io"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: "{{ .Release.Name }}-app"
port:
number: 80
status:
loadBalancer:
ingress:
- ip: 10.0.1.19
nginx-controller-service
kind: Service
metadata:
annotations:
external-dns.alpha.kubernetes.io/hostname: website.my-app.io, some.grafana.io
meta.helm.sh/release-name: nginx-ingress-controller
meta.helm.sh/release-namespace: default
finalizers:
- service.kubernetes.io/load-balancer-cleanup
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: nginx-ingress-controller
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx-ingress-controller
helm.sh/chart: nginx-ingress-controller-9.1.4
name: nginx-ingress-controller
namespace: default
resourceVersion: "997"
spec:
clusterIP: 172.30.0.140
clusterIPs:
- 172.30.0.140
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
nodePort: 30337
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 31512
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: nginx-ingress-controller
app.kubernetes.io/name: nginx-ingress-controller
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- hostname: someElbDns.eu-west-1.elb.amazonaws.com
Since your ingress controller service is of type LoadBalancer, creating this service will provision an NLB for you which will have your nginx controller pods as target groups.
A request will pass from the NLB to the ingress pods and then to grafana/your app subsequently.
I would try to use the same host (website.io) and the same nginx ingress class so both ingresses get assigned the same NLB address:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: default
name: grafana-ingress
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- grafana.website.io
secretName: tls-secret-website-io
rules:
- host: website.io
- http:
paths:
- path: /
backend:
serviceName: grafana-service
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: default
name: awebsite-ingress
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- subdomain.website.io
secretName: tls-secret-website-io
rules:
- host: website.io
- http:
paths:
- path: /(/|$)(.*)
backend:
serviceName: website-service
servicePort: 80
kind: Service
apiVersion: v1
metadata:
name: nginx-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: nginx-controller
app.kubernetes.io/part-of: nginx-controller
annotations:
# by default the type is elb (classic load balancer).
service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
# this setting is to make sure the source IP address is preserved.
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: nginx-controller
app.kubernetes.io/part-of: nginx-controller
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
Last step is to create a CNAME entry in Rote53 from your domain to the loadbalancer DNS.