I am trying to install the ingres-nginx-controller kubeadm (baremetal) , however for some reason it fails to start and when I try to apply my ingress rule it throws the below error:
Error from server (InternalError): error when creating "ing.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubern
etes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: dial tcp 10.103.2.38:44
3: connect: connection refused
I believed this was suggesting of unable to connect to the ingress-nginx-controller pods; so on checking I could see that the nginx controller pod was unable to start :
# kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-wwl67 0/1 CrashLoopBackOff 4 4m17s
pod/ingress-nginx-admission-patch-zclsr 0/1 CrashLoopBackOff 4 4m17s
pod/ingress-nginx-controller-75589bd5f6-hjk4z 0/1 ContainerCreating 0 4m27s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.96.192.255 <none> 80:30044/TCP,443:32048/TCP 4m27s
service/ingress-nginx-controller-admission ClusterIP 10.102.71.188 <none> 443/TCP 4m28s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 0/1 1 0 4m27s
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-75589bd5f6 1 1 0 4m27s
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 0/1 4m17s 4m27s
job.batch/ingress-nginx-admission-patch 0/1 4m17s 4m27s
I used the following to install the ingress-nginx:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.0/deploy/static/provider/baremetal/deploy.yaml
Also this [ https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml ] shows error 404
I tried to search but could not find any suggestive posts online. Please help me understand and resolve the issue.
Thanks.
Related
I am trying to enable the canary deployment for the AWS eks but my kayenta pod is not starting. When I describe the pod I see this error. Can anyone help?
Warning Unhealthy 12m (x2 over 12m) kubelet Readiness probe failed: wget: can't connect to remote host (127.0.0.1): Connection refused
Warning Unhealthy 2m56s (x59 over 12m) kubelet Readiness probe failed: wget: server returned error: HTTP/1.1 503
This is the status of pod:
NAME READY STATUS RESTARTS AGE
spin-clouddriver-d796bdc59-tpznw 1/1 Running 0 3h40m
spin-deck-77cc75b57d-w7rfp 1/1 Running 0 3h40m
spin-echo-db954bb9-phfd5 1/1 Running 0 3h40m
spin-front50-7c5684cf9-t7vl8 1/1 Running 0 3h40m
spin-gate-78d6779854-7xqz4 1/1 Running 0 3h40m
spin-kayenta-6d7b5fdfc6-p5tcp 0/1 Running 0 21m
spin-kayenta-869c46bfcf-8t5fh 0/1 Running 0 28m
spin-orca-7ddd66758d-mpnkg 1/1 Running 0 3h40m
spin-redis-5975cfcdc8-rnm9g 1/1 Running 0 45h
spin-rosco-b7dbb577-z4szz 1/1 Running 0 3h40m
I will try to address your issue from the Kubernetes perspective.
The errors you were experiencing:
Warning Unhealthy 12m (x2 over 12m) kubelet Readiness probe failed: wget: can't connect to remote host (127.0.0.1): Connection refused
Warning Unhealthy 2m56s (x59 over 12m) kubelet Readiness probe failed: wget: server returned error: HTTP/1.1 503
indicates that there was a problem with your ReadinessProbe configuration. Removing the ReadinessProbe from the deployment "fixed" the error but can cause some more issues in the future. To avoid that I recommend adding it back with a proper configuration:
Probes have a number of fields that you can use to more precisely
control the behavior of liveness and readiness checks:
initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to
0 seconds. Minimum value is 0.
periodSeconds: How often (in seconds) to perform the probe. Default to 10 seconds. Minimum value is 1.
timeoutSeconds: Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1.
successThreshold: Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1
for liveness. Minimum value is 1.
failureThreshold: When a probe fails, Kubernetes will try failureThreshold times before giving up. Giving up in case of liveness
probe means restarting the container. In case of readiness probe the
Pod will be marked Unready. Defaults to 3. Minimum value is 1.
You'll need to adjust the Probe's configuration based on your apps behavior (usually by trial and error). The two resources I would recommend that will help you with that are:
Configure Liveness, Readiness and Startup Probes
Kubernetes best practices: Setting up health checks with readiness and liveness probes
I am trying to deploy a specific version of ingress-controller with minikube and kubernetesv1.13, but from what I see it is only possible to have latest version of ingress-nginx-controller deployed.
I expect the ingress-nginx-controller-#####-#### pod to come back online and run with the nginx-ingress image version I point to in the deployments details.
After editing the ingress-nginx-controller deployment via kubectl edit and changing the image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller property from 0.32.0 to 0.24.1, the pod restarts and goes into CrashLoopBackOff state.
By hitting describe, the pod seems complaining about the node not having free ports:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m8s (x2 over 5m8s) default-scheduler 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
Normal Scheduled 4m54s default-scheduler Successfully assigned kube-system/ingress-nginx-controller-6c4b64d58c-s5ddz to minikube
After searching for a similar case I tried the following:
I check ss but see no port 80 or 443 being busy on the host:
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 32 192.168.122.1:53 0.0.0.0:*
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 5 127.0.0.1:631 0.0.0.0:*
LISTEN 0 5 [::1]:631 [::]:*
No pods seems to be in terminating status:
NAME READY STATUS RESTARTS AGE
coredns-86c58d9df4-7s55r 1/1 Running 1 3h14m
coredns-86c58d9df4-rtssn 1/1 Running 1 3h14m
etcd-minikube 1/1 Running 1 3h13m
ingress-nginx-admission-create-gpfml 0/1 Completed 0 47m
ingress-nginx-admission-patch-z96hd 0/1 Completed 0 47m
ingress-nginx-controller-6c4b64d58c-s5ddz 0/1 CrashLoopBackOff 9 24m
kube-apiserver-minikube 1/1 Running 0 145m
kube-controller-manager-minikube 1/1 Running 0 145m
kube-proxy-pmwxr 1/1 Running 0 144m
kube-scheduler-minikube 1/1 Running 0 145m
storage-provisioner 1/1 Running 2 3h14m
I did not create any yml file or custom deployment, just installed minikube and enabled the ingress addon.
How to use a different nginx-ingress-controller version ?
The Nginx Version is tied to minikube version.
First I tried previous versions. Unfortunatelly, the available Minikube v1.3 uses nginx 0.25.0 and Minikube v1.2 uses nginx 0.23.0
So the only way I found to run nginx 0.24.0 in Minikube was building the binary myself using minikube v1.4, here is the step-by-step:
Download the minikube 1.4 repository and extract it:
$ wget https://github.com/kubernetes/minikube/archive/v1.4.0.tar.gz
$ tar -xvzf v1.4.0.tar.gz
Then, cd into the newly created minikube-1.4.0 folder and edit the file deploy/addons/ingress/ingress-dp.yaml.tmpl changing the image version to 0.24.1 as below:
spec:
serviceAccountName: ingress-nginx
containers:
- name: controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
In order to build you have to download a Go distribution from original repo: https://golang.org/dl/
then follow the steps in https://golang.org/doc/install to install it. If you are running linux 64 bits, you can use the bellow comments:
$ wget https://dl.google.com/go/go1.14.4.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go1.14.4.linux-amd64.tar.gz
$ export PATH=$PATH:/usr/local/go/bin
Then from the Minikube 1.4.0 folder, run make:
/minikube-1.4.0$ ls
CHANGELOG.md CONTRIBUTING.md go.mod images Makefile OWNERS SECURITY_CONTACTS test.sh
cmd deploy go.sum installers netlify.toml pkg site third_party
code-of-conduct.md docs hack LICENSE README.md test translations
/minikube-1.4.0$ make
It may take a few minutes to download all dependencies, then let's copy the freshly build binary to /usr/local/bin and deploy minikube:
/minikube-1.4.0$ cd out/
/minikube-1.4.0/out$ ls
minikube minikube-linux-amd64
$ sudo cp minikube-linux-amd64 /usr/local/bin/minikube
$ minikube version
minikube version: v1.4.0
$ minikube start --vm-driver=kvm2 --kubernetes-version 1.13.12
NOTE: if you get an error about kvm2 driver when starting minikube, run the following command:
$ curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-kvm2 && sudo install docker-machine-driver-kvm2 /usr/local/bin/
This version comes with ingress enabled by default, let's check the deployment status:
$ minikube addons list | grep ingress
- ingress: enabled
$ kubectl describe deploy nginx-ingress-controller -n kube-system |
grep Image:
Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-54ff9cd656-d95w5 1/1 Running 0 2m14s
coredns-54ff9cd656-tnvnw 1/1 Running 0 2m14s
etcd-minikube 1/1 Running 0 78s
kube-addon-manager-minikube 1/1 Running 0 71s
kube-apiserver-minikube 1/1 Running 0 71s
kube-controller-manager-minikube 1/1 Running 0 78s
kube-proxy-wj2d6 1/1 Running 0 2m14s
kube-scheduler-minikube 1/1 Running 0 87s
nginx-ingress-controller-f98c6df-5h2l7 1/1 Running 0 2m9s
storage-provisioner 1/1 Running 0 2m8s
As you can see, the pod nginx-ingress-controller-f98c6df-5h2l7 is in running state.
If you have any question let me know in the comments.
Team,
I am able to fetch all pods running on a node with its namespace but my namespaces are generated dynamically and they change with characters in end. is there a way i can include a regex/pattern that I can use in kubectl command to pull all pods from all matching namespace?
kubectl get pods -n team-1-user1 --field-selector=spec.nodeName=node1,status.phase=Running
actual output1: works
NAMESPACE NAME READY STATUS RESTARTS AGE
team-1-user1 calico-node-9j5k2 1/1 Running 2 104d
team-1-user1 kube-proxy-ht7ch 1/1 Running 2 130d
I want below pulling pods for all namespaces starting with "team-".
kubectl get pods -n team-* --field-selector=spec.nodeName=node1,status.phase=Running
actual output2: fails
No resources found in team-workflow-2134-asf-324-d.yaml namespace.
expected outout: want this..
NAMESPACE NAME READY STATUS RESTARTS AGE
team-1-user1 calico-node-9j5k2 1/1 Running 2 104d
team-1-user1 kube-proxy-ht7ch 1/1 Running 2 130d
team-2-user1 calico-node-9j5k2 1/1 Running 2 1d
team-2-user1 kube-proxy-ht7ch 1/1 Running 2 10d
You can pipe the output of kubectl get pods into awk and match a regex for the same:
kubectl get pods --all-namespaces --no-headers | awk '{if ($1 ~ "team-") print $0}'
Here's a sample output for the same, searching for pods in kube- namespace:
❯❯❯ kubectl get pods --all-namespaces --no-headers | awk '{if ($1 ~ "kube-") print $0}'
kube-system coredns-6955765f44-27wxs 1/1 Running 0 107s
kube-system coredns-6955765f44-ztgq8 1/1 Running 0 106s
kube-system etcd-minikube 1/1 Running 0 109s
kube-system kube-addon-manager-minikube 1/1 Running 0 108s
I am attempting to deploy CockroachDB:v2.1.6 to a new AWS EKS cluster. Everything is deployed successfully; statefulset, services, pv's & pvc's are created. The AWS EBS volumes are created successfully too.
The issue is the pods never get to a READY state.
pod/cockroachdb-0 0/1 Running 0 14m
pod/cockroachdb-1 0/1 Running 0 14m
pod/cockroachdb-2 0/1 Running 0 14m
If I 'describe' the pods I get the following:
Normal Pulled 46s kubelet, ip-10-5-109-70.eu-central-1.compute.internal Container image "cockroachdb/cockroach:v2.1.6" already present on machine
Normal Created 46s kubelet, ip-10-5-109-70.eu-central-1.compute.internal Created container cockroachdb
Normal Started 46s kubelet, ip-10-5-109-70.eu-central-1.compute.internal Started container cockroachdb
Warning Unhealthy 1s (x8 over 36s) kubelet, ip-10-5-109-70.eu-central-1.compute.internal Readiness probe failed: HTTP probe failed with statuscode: 503
If I examine the logs of a pod I see this:
I200409 11:45:18.073666 14 server/server.go:1403 [n?] no stores bootstrapped and --join flag specified, awaiting init command.
W200409 11:45:18.076826 87 vendor/google.golang.org/grpc/clientconn.go:1293 grpc: addrConn.createTransport failed to connect to {cockroachdb-0.cockroachdb:26257 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup cockroachdb-0.cockroachdb on 172.20.0.10:53: no such host". Reconnecting...
W200409 11:45:18.076942 21 gossip/client.go:123 [n?] failed to start gossip client to cockroachdb-0.cockroachdb:26257: initial connection heartbeat failed: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup cockroachdb-0.cockroachdb on 172.20.0.10:53: no such host"
I came across this comment from the CockroachDB forum (https://forum.cockroachlabs.com/t/http-probe-failed-with-statuscode-503/2043/6)
Both the cockroach_out.log and cockroach_output1.log files you sent me (corresponding to mycockroach-cockroachdb-0 and mycockroach-cockroachdb-2) print out no stores bootstrapped during startup and prefix all their log lines with n?, indicating that they haven’t been allocated a node ID. I’d say that they may have never been properly initialized as part of the cluster.
I have deleted everything including pv's, pvc's & AWS EBS volumes through the kubectl delete command and reapplied with the same issue.
Any thoughts would be very much appreciated. Thank you
I was not aware that you had to initialize the CockroachDB cluster after creating it. I did the following to resolve my issue:
kubectl exec -it cockroachdb-0 -n /bin/sh
/cockroach/cockroach init
See here for more details - https://www.cockroachlabs.com/docs/v19.2/cockroach-init.html
After this the pods started running correctly.
When I start minikube and apply istio.yaml
bug the ingress can't start up:
eumji#eumji:~$ kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-ca-76dddbd695-bdwm9 1/1 Running 5 2d
istio-ingress-85fb769c4d-qtbcx 0/1 CrashLoopBackOff 67 2d
istio-mixer-587fd4bbdb-ldvhb 3/3 Running 15 2d
istio-pilot-7db8db896c-9znqj 2/2 Running 10 2d
When I try to see the log I get following output:
eumji#eumji:~$ kubectl logs -f istio-ingress-85fb769c4d-qtbcx -n istio-system
ERROR: logging before flag.Parse: I1214 05:04:26.193386 1 main.go:68] Version root#24c944bda24b-0.3.0-24ec6a3ac3a1d592d1873d2d8198278a849b8301
ERROR: logging before flag.Parse: I1214 05:04:26.193463 1 main.go:109] Proxy role: proxy.Node{Type:"ingress", IPAddress:"", ID:"istio-ingress-85fb769c4d-qtbcx.istio-system", Domain:"istio-system.svc.cluster.local"}
ERROR: logging before flag.Parse: I1214 05:04:26.193480 1 resolve.go:35] Attempting to lookup address: istio-mixer
ERROR: logging before flag.Parse: I1214 05:04:41.195879 1 resolve.go:42] Finished lookup of address: istio-mixer
Error: lookup failed for udp address: i/o timeout
Usage:
agent proxy [flags]
--serviceregistry string Select the platform for service registry, options are {Kubernetes, Consul, Eureka} (default "Kubernetes")
--statsdUdpAddress string IP Address and Port of a statsd UDP listener (e.g. 10.75.241.127:9125)
--zipkinAddress string Address of the Zipkin service (e.g. zipkin:9411)
Global Flags:
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
-v, --v Level log level for V logs (default 0)
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
ERROR: logging before flag.Parse: E1214 05:04:41.198640 1 main.go:267] lookup failed for udp address: i/o timeout
What could be the reason?
There is not enough information in your post to figure out what may be wrong, in particular it seems that somehow your ingress isn't able to resolve istio-mixer which is unexpected.
Can you file a detailed issue
https://github.com/istio/issues/issues/new
And we can take it from there ?
Thanks
Are you using something like minikube? The quick-start docs give this hint: "Note: If your cluster is running in an environment that does not support an external load balancer (e.g., minikube), the EXTERNAL-IP of istio-ingress says . You must access the application using the service NodePort, or use port-forwarding instead."
https://istio.io/docs/setup/kubernetes/quick-start.html