I installed Kubernetes ingress controller on GKE following the official documentation as following.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/cloud/deploy.yaml
The ingress controller runs fine.
ingress-nginx-admission-create-dvkgp 0/1 Completed 0 5h29m
ingress-nginx-admission-patch-58l4z 0/1 Completed 1 5h29m
ingress-nginx-controller-65d7564f46-2rtjs 1/1 Running 0 5h29m
It creates a TCP load balancer, health checkup and firewall rules automatically. My kubernetes cluster has 3 nodes. Interestingly, the health checkup fails for 2 instances. It passes for the instance where the ingress controller is running. I debug it but didn't find any clue. Could someone help me on this.
If you were to look into the deploy.yaml you applied you would see:
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
type: LoadBalancer
externalTrafficPolicy: Local
Notice the externalTrafficPolicy: Local. It is being used to Preserve the client source ip.
It's even better explained here: Source IP for Services with Type=LoadBalancer
From k8s docs:
However, if you're running on Google Kubernetes Engine/GCE, setting the same service.spec.externalTrafficPolicy field to Local forces nodes without Service endpoints to remove themselves from the list of nodes eligible for loadbalanced traffic by deliberately failing health checks.
These health checkups are designed to fail. It works that way so that client IPs can be preserved.
Notice that the one node that is listed as healthy is the one where ingress-nginx-controller pod runs. Delete this pod and wait for it to reschedule on a different node - now this other node should be healthy. Now run 3 pod replicas, one on every node and all nodes will be healthy.
One of the possible reason is Firewall rules. Google has specified IP range and port details of Google Health Check probers. You have to configure ingress allow rule to establish health check probing connection to your backend.
For additional debugging details check this Google Cloud Platform blog: Debugging Health Checks in Load Balancing on Google Compute Engine
Related
Once a week the load balancer stops communicating with the workloads. the load balancer stops having communication with the workloads.
On the one hand I can see that the Workload was restarted since I see the last restart time, to solve this what I do is restart the Workloads manually and there is a connection again,
My question is, why does it restart? why if it restarts we leave the loadBalancer without communication with the Worckloads?
More info:
The loadBalancer is of type Internal HTTP(S), with respect to the provisioning we do it from a yaml file deployed in kubernetes:
# internal-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ilb-cym-ingress
namespace: k8s-cym
annotations:
kubernetes.io/ingress.class: "gce-internal"
The problem is: for some reason once in a while (1 week, 5 days ...) we receive an alert that we have configured when our WorkLoad is restarted, this causes in the Ingress that the BackEnd referring to this WorkLoad remains in UnHealthy state.
On the other hand, we can redeploy from the GCP console and we did not find any problems.
Regarding the configuration of the HealthCheck we have an interval of 60 sec with a timeout of 30 sec.
And in terms of the Threshold we have the Healthy in 1 attempt and the unHealthhy in 5 attempts, these parameters we have been varying to see if we managed to solve it but we have not managed to solve it.
Finally, I wanted to comment that when the workload is launched we have an initial delay of 20 sec to allow time for the database to connect correctly. I don't know if this can interfere with our HealtChecks
NOTE: I tried to include screenshots but stackoverflow does not allow me to add images with preview so I included them as links.
I deployed a web app on AWS using kOps.
I have two nodes and set up a Network Load Balancer.
The target group of the NLB has two nodes (each node is an instance made from the same template).
Load balancer actually seems to be working after checking ingress-nginx-controller logs.
The requests are being distributed over pods correctly. And I can access the service via ingress external address.
But when I go to AWS Console / Target Group, one of the two nodes is marked as and I am concerned with that.
Nodes are running correctly.
I tried to execute sh into nginx-controller and tried curl to both nodes with their internal IP address.
For the healthy node, I get nginx response and for the unhealthy node, it times out.
I do not know how nginx was installed on one of the nodes and not on the other one.
Could anybody let me know the possible reasons?
I had exactly the same problem before and this should be documented somewhere on AWS or Kubernetes. The answer is copied from AWS Premium Support
Short description
The NGINX Ingress Controller sets the spec.externalTrafficPolicy option to Local to preserve the client IP. Also, requests aren't routed to unhealthy worker nodes. The following troubleshooting implies that you don't need to maintain the cluster IP address or preserve the client IP address.
Resolution
If you check the ingress controller service you will see the External Traffic Policy field set to Local.
$ kubectl -n ingress-nginx describe svc ingress-nginx-controller
Output:
Name: ingress-nginx-controller
Namespace: ingress-nginx
...
External Traffic Policy: Local
...
This Local setting drops packets that are sent to Kubernetes nodes that aren't running instances of the NGINX Ingress Controller. Assign NGINX pods (from the Kubernetes website) to the nodes that you want to schedule the NGINX Ingress Controller on.
Update the pec.externalTrafficPolicy option to Cluster
$ kubectl -n ingress-nginx patch service ingress-nginx-controller -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
Output:
service/ingress-nginx-controller patched
By default, NodePort services perform source address translation (from the Kubernetes website). For NGINX, this means that the source IP of an HTTP request is always the IP address of the Kubernetes node that received the request. If you set a NodePort to the value of the externalTrafficPolicy field in the ingress-nginx service specification to Cluster, then you can't maintain the source IP address.
E.g. an istio service
istio-ingressgateway LoadBalancer 10.103.19.83 10.160.32.41 15021:30943/TCP,80:32609/TCP,443:30341/TCP,3306:30682/TCP,15443:30302/TCP
Which resulted in a TCP internal load balancer. The front end is ports 15021, 80, 443, 3306, and 15443.
The backend is basically the instance group of the cluster.
How does the load balancer know 443 at the front end will forward to 30341 at backend? As far as I know, TCP load balancer is doing port forwarding? How/Where does the magic happening
The LoadBalancer Service type is an extension of the NodePort type, which is an extension of the ClusterIP type. A nodePort just opens up a port in the range 30000-32767 on each worker node and uses a label selector to identify which Pods to send the traffic to.
This means that internal clients call the Service by using the internal IP address of a node along with the TCP port specified by nodePort. The request is forwarded to one of the member Pods on the TCP port specified by the targetPort field.
Here’s an example
When a Service is created in kubernetes, a corresponding Endpoints object is created along with it. It also applies to LoadBalancer service type.
If you create a simple nginx deployment e.g. by running:
kubectl apply -f https://k8s.io/examples/application/deployment.yaml
and then expose it as a LoadBalancer service:
kubectl apply -f https://k8s.io/examples/application/deployment.yaml
apart from the service itself, you will also see the lb-nginx Endpoints object. You can inspect its details:
kubectl get ep lb-nginx -o yaml
As you can see it keeps track of all exposed pods (being part of a Deployment in this case) so that corresponding iptables rules, which are responsible for forwarding the traffic to a particular pod, can be up-to-date all the time, even if number of them or their ip chages.
You can e.g. scale your deployment to 5 replicas:
kubectl scale deployment nginx-deployment --replicas=5
and inspect the Endpoints object again:
kubectl get ep lb-nginx -o yaml
and you will see that right after your 5 pods are up and running it immediately gets updated as well.
As you can see in subsets section of the yaml:
subsets:
- addresses:
- ip: 10.12.0.3
nodeName: gke-gke-default-pool-75259266-oauz
targetRef:
kind: Pod
name: nginx-deployment-66b6c48dd5-dw9mt
namespace: default
resourceVersion: "22394113"
uid: 8d7e1d3e-64e2-4891-b567-61ee48f61ed1
apart from the ip address of the Pod it maintains information about the node on which it is running.
Let's go back for a moment to the Service:
kubectl get svc lb-nginx -o yaml
As you can see LoadBalancer service apart from its external IP address has its ClusterIP as every other Service (well, almost every as headless services don't have ClusterIP):
spec:
clusterIP: 10.16.6.236
clusterIPs:
- 10.16.6.236
externalTrafficPolicy: Cluster
ports:
- nodePort: 31935
port: 80
protocol: TCP
targetPort: 80
So as you can imagine this external IP is somehow mapped to the cluster ip so it route the traffic further to the respective endpoints in the cluster. How exactly this mapping is done doesn't really matter as it is done by the cloud provider and such implementation details are not part of publicly shared knowledge. The only thing you need to know is that when your cloud provider provisions an external load balancer to satisfy your request defined in a Service of LoadBalancer type, apart from creating an external load balancer it takes care of the mapping between this external IP and some standard port assigned to it and a kubernetes service which has all the information needed to route the traffic further to the respective pods. In case you wonder how exactly this is done on GCP side i.e. mapping/binding between the external (or internal) loadbalancer and kubernetes LoadBalancer service, I'm affraid such implementation details are not publicly revealed.
I'm trying to create a TCP/UDP load balancer on GCP to allow HA on my service, but I've noticed that when I create a destination group, all instances in that group are marked as unhealthy and are not being checked by google (I've seen the machine logs to check it). The firewall is open because is for testing purpose, so I'm sure that is not the problem.
I've created an HTTP/S load balancer using a backend with similar check configuration and the same machine is marked as healthy, so is not a problem of that machine (even now the logs shows how google is really checking that instance).
Both checks are HTTP to port 80, so I'm not able to see where's the problem and the difference between both kind of load balancers checkers.
Also I've checked to disable health check but the instance still marked as unhealthy and the traffic is not being sent to any of the instances, so the load balancer is not usefull it all.
Is necessary any other configuration to make it check the instance?
Thanks and greetings!!
Creating a TCP load balancer
When you're using any of the Google Cloud load balancers, you need not expose your VM's external ports to the internet, only your load balancer needs to be able to reach it.
The steps to create a TCP load balancer are described here. I find it convenient to use gcloud and run the commands, but you can also use the Cloud Console UI to achieve the same result.
I tried the below steps and it works for me (you can easily modify this to make it work with UDP as well - remember you still need HTTP health checks even when using UDP load balancing):
# Create 2 new instances
gcloud compute instances create vm1 --zone us-central1-f
gcloud compute instances create vm2 --zone us-central1-f
# Make sure you have some service running on port 80 on these VMs after creation.
# Create an address resource to act as the frontend VIP.
gcloud compute addresses create net-lb-ip-1 --region us-central1
# Create a HTTP health check (by default uses port 80).
$ gcloud compute http-health-checks create hc-1
# Create a target pool associated with the health check you just created.
gcloud compute target-pools create tp-1 --region us-central1 --http-health-check hc-1
# Add the instances to the target pool
gcloud compute target-pools add-instances tp-1 --instances vm1,vm2 --instances-zone us-central1-f
# Create a forwarding rule associated with the frontend VIP address we created earlier
# which will forward the traffic to the target pool.
$ gcloud compute forwarding-rules create fr-1 --region us-central1 --ports 80 --address net-lb-ip-1 --target-pool tp-1
# Describe the forwarding rule
gcloud compute forwarding-rules describe fr-1 --region us-central1
IPAddress: 1.2.3.4
IPProtocol: TCP
creationTimestamp: '2017-07-19T10:11:12.345-07:00'
description: ''
id: '1234567890'
kind: compute#forwardingRule
loadBalancingScheme: EXTERNAL
name: fr-1
portRange: 80-80
region: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/regions/us-central1
selfLink: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/regions/us-central1/forwardingRules/fr-1
target: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/regions/us-central1/targetPools/tp-1
# Check the health status of the target pool and verify that the
# target pool considers the backend instances to be healthy
$ gcloud compute target-pools get-health tp-1
---
healthStatus:
- healthState: HEALTHY
instance: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/zones/us-central1-f/instances/vm1
ipAddress: 1.2.3.4
kind: compute#targetPoolInstanceHealth
---
healthStatus:
- healthState: HEALTHY
instance: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/zones/us-central1-f/instances/vm2
ipAddress: 1.2.3.4
kind: compute#targetPoolInstanceHealth
HTTP Health Checks are required for non-proxy TCP/UDP load balancers
If you're using a UDP load balancer (which is considered Network Load Balancing in Google CLoud), you will need to spin up a basic HTTP server which can respond to HTTP health checks in addition to your service which is listening on a UDP port for incoming traffic.
The same also applies to non-proxy based TCP load balancers (which is also considered Network Load balancing in Google Cloud).
This is documented here.
Health checking
Health checks ensure that Compute Engine forwards new connections only
to instances that are up and ready to receive them. Compute Engine
sends health check requests to each instance at the specified
frequency; once an instance exceeds its allowed number of health check
failures, it is no longer considered an eligible instance for
receiving new traffic. Existing connections will not be actively
terminated which allows instances to shut down gracefully and to close
TCP connections.
The health check continues to query unhealthy instances, and returns
an instance to the pool once the specified number of successful checks
is met.
Network load balancing relies on legacy HTTP Health checks for
determining instance health. Even if your service does not use HTTP,
you'll need to at least run a basic web server on each instance that
the health check system can query.
Some info:
Kubernetes (1.5.1)
AWS
1 master and 1 node (both ubuntu 16.04)
k8s installed via kubeadm
Terraform made by me
Please don't reply use kube-up, kops or similar. This is about understanding how k8s works under the hood. There is by far too much unexplained magic in the system and I want to understand it.
== Question:
When creating a Service of type load balancer on k8s[aws] (for example):
apiVersion: v1
kind: Service
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-addon: kubernetes-dashboard.addons.k8s.io
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: "true"
facing: external
spec:
type: LoadBalancer
selector:
k8s-app: kubernetes-dashboard
ports:
- port: 80
I successfully create an internal or external facing ELB but none of the machines are added to the ELB (I can taint the master too but nothing changes). My problem is basically this:
https://github.com/kubernetes/kubernetes/issues/29298#issuecomment-260659722
The subnets and nodes (but not the VPC) are all tagged with "KubernetesCluster" (again... elb are created in the right place). However no nodes is added.
In the logs
kubectl logs kube-controller-manager-ip-x-x-x-x -n kube-system
after:
aws_loadbalancer.go:63] Creating load balancer for
kube-system/kubernetes-dashboard with name:
acd8acca0c7a111e69ca306f22de69ae
There is no other output (it should print the nodes added or removed). I tried to understand the code at:
https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws_loadbalancer.go
But whatever is the reason, this function to not add nodes.
The documentation doesn't go at length trying to explain the "process" behind k8s decisions. To try to understand k8s I tried/used kops, kube up, kubeadm, kubernetes the hard way repo and reading damn code, but still I am unable to understand how k8s on aws SELECTS the node to add to the elb.
As a consequence, also no security group is changed anywhere.
Is it a tag on the ec2?
Kublet setting?
Anything else?
Any help is greatly appreciated.
Thanks,
F.
I think Steve is on the right track. Make sure your kubelets, apiserver, and controller-manager components all include --cloud-provider=aws in their arguments lists.
You mention your subnets and instances all have matching KubernetesCluster tags. Do your controller & worker security groups? K8s will modify the worker SG in particular to allow traffic to/from the service ELBs it creates. I tag my VPC as well, though I guess it's not required and may prohibit another cluster from living in the same VPC.
I also tag my private subnets with kubernetes.io/role/internal-elb=true and public ones with kubernetes.io/role/elb=true to identify where internal and public ELBs can be created.
The full list (AFAIK) of tags and annotations lives in https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go
I think the node registration is being managed outside of Kubernetes. I'm using kops and if I edit the size of my ASG in AWS the new nodes are not registered with my service ELBs. But if I edit the number of nodes using kops the new nodes are there.
In the docs a kops instance group maps to an ASG when running on AWS. In the code it looks like its calling AWS rather than a k8s API.
I know you're not using kops but I think in Terraform you need to replicate the AWS API calls that kops is making.
Make sure you are setting the correct cloud provider settings with kubeadm (http://kubernetes.io/docs/admin/kubeadm/).
The AWS cloud provider automatically syncs the nodes available with the ELB. I created an type LoadBalancer then scaled my cluster and the new node was eventually added the ELB: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws_loadbalancer.go#L376