I am new to the AWS EKS and I want to know how I can setup an ingress and enable TLS (with a free service such as lets-encrypt).
I have deployed an EKS cluster and I have the following sample nginx manifest.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service-loadbalancer
spec:
type: LoadBalancer. // <------ can't I use a ClusterIp and still have a LB priovisioned?
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
---
#05-ALB-Ingress-Basic.yml
# Annotations Reference: https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/annotation/
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-usermgmt-restapp-service
labels:
app: usermgmt-restapp
annotations:
# Ingress Core Settings
kubernetes.io/ingress.class: "alb"
alb.ingress.kubernetes.io/scheme: internet-facing
# Health Check Settings
alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
alb.ingress.kubernetes.io/healthcheck-path: /usermgmt/health-status
alb.ingress.kubernetes.io/healthcheck-interval-seconds: '15'
alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '5'
alb.ingress.kubernetes.io/success-codes: '200'
alb.ingress.kubernetes.io/healthy-threshold-count: '2'
alb.ingress.kubernetes.io/unhealthy-threshold-count: '2'
spec:
rules:
- http:
paths:
- path: /*
pathType: Prefix
backend:
service:
name: nginx-service-loadbalancer
port:
number: 80
When it creates the LoadBalancer type service, it go ahead and creates a classic load balancer.
My questions are:
How can I provision (automatically) a Layer7 application load balancer and not the classic load balancer
Instead of using LoadBalancer type service, can I use a ClusterIP service and use my ingress to point to that and still create an automatic Load Balancer?
Thank you!
How can I provision (automatically) a Layer7 application load
balancer and not the classic load balancer
By using an ingress resource and specifying kubernetes.io/ingress.class: "alb".
Instead of using LoadBalancer type service, can I use a ClusterIP service and use my ingress to point to that and still
create an automatic Load Balancer?
yes, when used alb ingress resource with annotation alb.ingress.kubernetes.io/target-type: ip you can use a clusterip service.
so please don't create both a service-type loadbalancer and ingress resource at the same time.
Related
I have an EKS kubernetes cluster with AWS Load Balancer Controller and Argo CD installed. I'm creating an Application Load Balancer based on Argo CD documentation here.
Basically, I'm creating a NodePort service that receives traffic from the load balancer, and an ingress that will create the load balancer (using AWS Load Balancer Controller).
The ingress code is this one:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/backend-protocol: HTTPS
alb.ingress.kubernetes.io/backend-protocol-version: HTTP2
# Use this annotation (which must match a service name) to route traffic to HTTP2 backends.
alb.ingress.kubernetes.io/conditions.argogrpc: |
[{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "Content-Type", "values":["application/grpc"]}}]
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
# ALB annotations
kubernetes.io/ingress.class: 'alb'
alb.ingress.kubernetes.io/scheme: 'internet-facing'
alb.ingress.kubernetes.io/target-type: 'instance'
alb.ingress.kubernetes.io/load-balancer-name: 'test-argocd'
alb.ingress.kubernetes.io/certificate-arn: 'arn:aws:acm:us-east-1:1234567:certificate/longcertcode'
alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true
# Health Check
alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
name: argocd
namespace: argocd
spec:
rules:
- host: argocd.argoproj.io
http:
paths:
- path: /
backend:
service:
name: argogrpc
port:
number: 443
pathType: ImplementationSpecific
tls:
- hosts:
- argocd.argoproj.io
defaultBackend:
service:
name: argogrpc
port:
number: 443
And that creates a Load Balancer as expected.
I'm creating the service with this:
apiVersion: v1
kind: Service
metadata:
annotations:
alb.ingress.kubernetes.io/backend-protocol-version: HTTP2
labels:
app: argogrpc
name: argogrpc
namespace: argocd
spec:
ports:
- name: "443"
port: 443
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: argocd-server
sessionAffinity: None
type: NodePort
The issue here is that the health check is failing on the Target Group:
If I change the backend protocol version to GRPC:
apiVersion: v1
kind: Service
metadata:
annotations:
alb.ingress.kubernetes.io/backend-protocol-version: GRPC
labels:
app: argogrpc
name: argogrpc
namespace: argocd
spec:
ports:
- name: "443"
port: 443
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: argocd-server
sessionAffinity: None
type: NodePort
Then the health check is passed, but I get an 464 error on Chrome:
This is what AWS documentation says about this error, but it doesn't help to clarify why I'm getting it:
So the question is, how do I create an application load balancer for my Argo CD using AWS load balancer controller that works? According to the documentation, it should work in both cases.
I am running a simple web application inside pods and have mapped them with load balaner. I was able to curl it from the ec2 machine but couldn't access it from outside. Am I missing something in configuration?. Here is my deployment and service yml.
Service
apiVersion: v1
kind: Service
metadata:
name: load-balancer-service
spec:
type: LoadBalancer
selector:
tag: frontend
ports:
- name: port-lb-k8s
protocol: TCP
port: 8080
targetPort: 80
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend-deployment
spec:
selector:
matchLabels:
tag: frontend
replicas: 3 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
tag: frontend
spec:
containers:
- name: frontend-container
image: coitlearning/coit-frontend
Ec2 Machine
In-order to create a service with an internet-facing Network Load Balancer that load balances to IP targets, you can use the following
apiVersion: v1
kind: Service
metadata:
name: nlb-load-balancer-service
namespace: default
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
ports:
- port: 8080
targetPort: 80
protocol: TCP
type: LoadBalancer
selector:
tag: frontend
You can get more details in official-docs
I have AWS K3S Kubernetes Cluster
I have AWS Load Balancer
I have registered domain
I have registered AWS Certificate
I created CNAME record for my domain and for AWS Load Balancer DNS name
I installed Traefik Ingress Controller on AWS K3S Kubernetes Cluster
I deployed "usermgmt" and "whoami" services to AWS K3S Kubernetes Cluster
I created Traefik Ingress with paths to "usermgmt" and "whoami"
The question is:
How to connect my AWS Load Balancer, which is hosted on my domain, to my services on K3s, using Ingress Traefik Controller?
Or in other words:
How to adapt "traefik service" or "traefik deployment", described below, to use AWS Certificate Resolver for my registered domain?
Or any example of how to use
AWS Load Balancer, AWS Target Group, AWS Security Group, created with Terraform files
in combination with Traefik Ingress Controller and Traefik Ingress Routes, deployed to K3S Kubernetes Cluster, resolved with AWS Certificate.
I currently can't connect to my services through AWS Load Balancer.
The following errors are returned:
404 Page Not Found
502 Bad Gateway
Here are the examples of URLs, which I try:
https://keycloak.skycomposer.net/usermgmt
https://keycloak.skycomposer.net/whoami
I set up correspondent Ingress Routes for "usermgmt" and "whoami" kubernetes services.
Here is some more information:
I created K3S Kubernetes Cluster in AWS with Load Balancer
These are my terraform files:
https://github.com/skyglass/user-management/tree/master/terraform
K3S cluster is deployed to EC2 instance (see "userdata.tpl" script)
I disabled Traefik Ingress Controller deployment, so I could deploy it later.
I found example on how to install "Traefik" to K3S Kubernetes cluster here:
https://github.com/sleighzy/k3s-traefik-v2-kubernetes-crd
Unfortunately, this example uses "godaddy" certificate resolver, but my domain is registered with AWS Route 53 and I use AWS certificate manager.
Here are files for "traefik service" and "traefik deployment", which I try to adapt:
traefik-service:
---
apiVersion: v1
kind: Service
metadata:
name: traefik
namespace: kube-system
spec:
# The targetPort entries are required as the Traefik container is listening on ports > 1024
# so that the container can be run as a non-root user and they can bind to these ports.
# Traefik is still accessed over 80 and 443 on the host, but the service routes the traffic
# to ports 8080 and 8443 on the container.
ports:
- protocol: TCP
name: web
port: 80
targetPort: 8080
- protocol: TCP
name: websecure
port: 443
targetPort: 8443
- protocol: TCP
name: admin
port: 8080
targetPort: 9080
selector:
app: traefik
# Set externalTrafficPolicy to Local so that all external traffic intended for
# the Traefik pod goes directly to that local node. If the default of Cluster is
# used instead then the client source IP address is lost, and may hop between nodes.
externalTrafficPolicy: Local
type: LoadBalancer
traefik-deployment:
---
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: kube-system
name: traefik-ingress-controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: kube-system
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.4
args:
- --api.dashboard=true
- --ping=true
- --accesslog
- --entrypoints.traefik.address=:9080
- --entrypoints.web.address=:8080
- --entrypoints.websecure.address=:8443
# Uncomment the below lines to redirect http requests to https.
# This specifies the port :443 and not the https entrypoint name for the
# redirect as the service is listening on port 443 and directing traffic
# to the 8443 target port. If the entrypoint name "websecure" was used,
# instead of "to=:443", then the browser would be redirected to port 8443.
- --entrypoints.web.http.redirections.entrypoint.to=:443
- --entrypoints.web.http.redirections.entrypoint.scheme=https
- --providers.kubernetescrd
- --providers.kubernetesingress
- --certificatesresolvers.myresolver.acme.tlschallenge=true
- --certificatesresolvers.myresolver.acme.email=postmaster#example.com
- --certificatesresolvers.myresolver.acme.storage=/etc/traefik/certs/acme.json
# Please note that this is the staging Let's Encrypt server.
# Once you get things working, you should remove that whole line altogether.
# - --certificatesresolvers.godaddy.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
- --log
- --log.level=INFO
livenessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: 9080
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
resources:
limits:
memory: '100Mi'
cpu: '1000m'
ports:
# The Traefik container is listening on ports > 1024 so the container
# can be run as a non-root user and they can bind to these ports.
- name: web
containerPort: 8080
- name: websecure
containerPort: 8443
- name: admin
containerPort: 9080
volumeMounts:
- name: certificates
mountPath: /etc/traefik/certs
# volumes:
# - name: certificates
# persistentVolumeClaim:
# claimName: traefik-certs-pvc
volumes:
- name: certificates
hostPath:
path: "/Users/dddd/git/aws/letsencrypt:/etc/traefik/certs"
See other files here: https://github.com/sleighzy/k3s-traefik-v2-kubernetes-crd
Ideally there should be solution like this:
apiVersion: v1
kind: Service
metadata:
name: traefik-proxy
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:REGION:ACCOUNTID:certificate/CERT-ID"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
spec:
type: LoadBalancer
selector:
app: traefik-proxy
tier: proxy
ports:
- port: 443
targetPort: 80
In this solution, I would just provide my AWS Certificate ARN and traefik ingress controller will do everything else.
The similar solution is described in this article:
https://www.ronaldjamesgroup.com/blog/getting-started-with-traefik
But, unfortunately, this solution doesn't work for me too, I tried it without any success.
The following errors are returned:
404 Page Not Found
502 Bad Gateway
when I try Ingress Route Paths for my domain:
https://keycloak.skycomposer.net/usermgmt
https://keycloak.skycomposer.net/whoami
After trying several options, I finally found the solution:
https://github.com/skyglass-examples/aws-k3s-traefik
I created AWS Load Balancer and K3S cluster with Terraform
I created Traefik Ingress Controller kubernetes manifest files
I created kubernetes manifest files for 2 services
I registered AWS Load Balancer DNS name for my domain
I created AWS Certificate for my domain
I used AWS Certificate ARN for Traefik Ingress Controller and AWS HTTPS Load Balancer
Here are my Traefik Ingress Controller manifest files:
traefik-deployment.yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik-ingress-controller
namespace: kube-system
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: traefik-proxy
namespace: kube-system
labels:
app: traefik-proxy
tier: proxy
spec:
replicas: 1
selector:
matchLabels:
app: traefik-proxy
tier: proxy
template:
metadata:
labels:
app: traefik-proxy
tier: proxy
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
containers:
- image: traefik:v1.2.0-rc1-alpine
name: traefik-proxy
ports:
- containerPort: 80
hostPort: 80
name: traefik-proxy
- containerPort: 8080
name: traefik-ui
args:
- --web
- --kubernetes
traefik-service.yaml:
apiVersion: v1
kind: Service
metadata:
name: traefik-proxy
namespace: kube-system
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-1:dddddddddd"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
service.beta.kubernetes.io/aws-load-balancer-type: "alb"
spec:
type: LoadBalancer
externalTrafficPolicy: Local
selector:
app: traefik-proxy
tier: proxy
ports:
- port: 443
targetPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
selector:
app: traefik-proxy
tier: proxy
ports:
- port: 80
targetPort: 8080
traefik-ingress.yaml:
apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
name: traefik-lb
spec:
controller: traefik.io/ingress-controller
---
apiVersion: "networking.k8s.io/v1beta1"
kind: "Ingress"
metadata:
name: "traefik-usermgmt-ingress"
spec:
ingressClassName: "traefik-lb"
rules:
- host: "keycloak.skycomposer.net"
http:
paths:
- path: "/usermgmt"
backend:
serviceName: "usermgmt"
servicePort: 80
---
apiVersion: "networking.k8s.io/v1beta1"
kind: "Ingress"
metadata:
name: "traefik-whoami-ingress"
spec:
ingressClassName: "traefik-lb"
rules:
- host: "keycloak.skycomposer.net"
http:
paths:
- path: "/whoami"
backend:
serviceName: "whoami"
servicePort: 80
See the full code here:
https://github.com/skyglass-examples/aws-k3s-traefik
The code includes:
terraform files for AWS Load Balancer and K3S Kubernetes Cluster
source code for one of the docker containers, which I deployed to K3S
kubernetes manifest files for Traefik Ingress Controller, 2 Kubernetes Services and Traefik Ingress, which exposes these services with secured HTTPS connection on registered domain.
Replace AWS Certificate ARN with correspondent ARN of your certificate
Replace "skycomposer.net" with your domain name (see more details in the Readme file: https://github.com/skyglass-examples/aws-k3s-traefik)
I am working on Kubernetes where our pods are running on AWS. I am creating a service with pre-defined LoadBalancer specifications. Still, Kubernetes is adding an extra SG to the loadbalancer. How can I specify not to do that? Thank you.
service.yaml :
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: akeneo-service
app.kubernetes.io/instance: akeneo-service-instance
app.kubernetes.io/version: "1.0.0"
app.kubernetes.io/component: backend
app.kubernetes.io/managed-by: kubectl
annotations:
# Note that the backend talks over HTTP.
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "ssl"
# TODO: Fill in with the ARN of your certificate.
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: OUR_ARN
# Only run SSL on the port named "https" below.
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: "sg-0d3a65fc39e47e3cf"
name: akeneo-service
spec:
selector:
app: akeneo-service
type: LoadBalancer
ports:
- name: https
port: 443
targetPort: 443
- name: http
port: 80
targetPort: 80
Any help would be nice. :-)
After I created a traefik daemonset, I created a service as loadbalancer on port 80, which is the Traefik proxy port and the node got automatically registered to it. If i hit the elb i get the proxy 404 which is OK because no service is registered yet
I then created a nodeport service for the web-ui. targeted port 8080 inside the pod and 80 on clusterip. I can curl the traefik ui from inside the cluster and it retruns traefik UI
I then created an ingress so that when i hit elb/ui it gets me to the backend web-ui service of traefik and it fails. I also have the right annotations in my ingress but the elb does not route the path to the traefik ui in the backend which is running properly
What am i doing wrong here? I can post all my yml files if required
UPD
My yaml files:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: traefik
labels:
app: traefik
spec:
template:
metadata:
labels:
app: traefik
spec:
containers:
- image: traefik
name: traefik
args:
- --api
- --kubernetes
- --logLevel=INFO
- --web
ports:
- containerPort: 8080
name: traefikweb
- containerPort: 80
name: traefikproxy
apiVersion: v1
kind: Service
metadata:
name: traefik-proxy
spec:
selector:
app: traefik
ports:
- port: 80
targetPort: traefikproxy
type: LoadBalancer
apiVersion: v1
kind: Service
metadata:
name: traefik-web-ui
spec:
selector:
app: traefik
ports:
- name: http
targetPort: 8080
nodePort: 30001
port: 80
type: NodePort
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: default
name: traefik-ing
annotations:
kubernetes.io/ingress.class: traefik
#traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip:/ui
spec:
rules:
- http:
paths:
- path: /ui
backend:
serviceName: traefik-web-ui
servicePort: 80
If you are on Private_Subnets use
kind: Service
metadata:
name: traefik-proxy
> annotations:
> "service.beta.kubernetes.io/aws-load-balancer-internal": "0.0.0.0/0"
spec:
selector:
app: traefik
ports:
- port: 80
targetPort: traefikproxy
type: LoadBalancer```
I then created an ingress so that when i hit elb/ui it gets me to the backend web-ui service of traefik and it fails."
How did it fail? Did you get error 404, error 500, or something else?
Also, for traefik-web-ui service, you don't need to set type: NodePort, it should be type: ClusterIP.
When you configure backends for your Ingress, the only requirement is availability from inside a cluster, so ClusterIP type will be more than enough for that.
Your service should be like that:
apiVersion: v1
kind: Service
metadata:
name: traefik-web-ui
spec:
selector:
app: traefik
ports:
- name: http
targetPort: 8080
port: 80
Option PathPrefixStrip can be useful because without it request will be forwarded to UI with /ui prefix, which you definitely don't want.
All other configs look good.