After spending half a day digging the web, I'm still not able to find the reason why my worker nodes are not getting proper labels. I have my worker nodes in Auto scaling group in AWS and I'm using AWS cloud provider with Kubeadm to provision my cluster. After all my master nodes and worker nodes come up, I don't see proper labels like availability zone assigned to worker nodes which are present in the master nodes as seen below:
# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
ip-10-100-128-9.ec2.internal Ready <none> 5h12m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-128-9.ec2.internal,kubernetes.io/os=linux
ip-10-100-148-5.ec2.internal Ready <none> 5h12m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-148-5.ec2.internal,kubernetes.io/os=linux
ip-10-100-164-199.ec2.internal Ready <none> 5h12m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-164-199.ec2.internal,kubernetes.io/os=linux
ip-10-100-3-145.ec2.internal Ready master 5h15m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1c,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-3-145.ec2.internal,kubernetes.io/os=linux,node-role.kubernetes.io/master=
ip-10-100-3-55.ec2.internal Ready master 5h15m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1a,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-3-55.ec2.internal,kubernetes.io/os=linux,node-role.kubernetes.io/master=
ip-10-100-3-76.ec2.internal Ready master 5h15m v1.14.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1b,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-100-3-76.ec2.internal,kubernetes.io/os=linux,node-role.kubernetes.io/master=
I somehow doubt this to be the reason why I can't create a PVC for my PV which I want to attach to my docker registry pod.
# kubectl describe pod docker-registry-5b66dd644d-66f7k -n default | grep -A4 -i Events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 84s (x11 over 14m) default-scheduler 0/6 nodes are available: 3 node(s) had taints that the pod didn't tolerate, 3 node(s) had volume node affinity conflict.
Warning FailedScheduling 79s (x3 over 82s) default-scheduler 0/4 nodes are available: 4 node(s) had taints that the pod didn't tolerate.
I see my storageclass and PV created and can verify the same volume ID on AWS. I also manually tried attaching a volume to my worker nodes using AWS CLI where I was successful but not through PVC. Please advise what's getting wrong here. Thanks in Advance.
The Issue was with the kubelet service file. By default, kubeadm doesn't add AWS cloud provider information in the kubelet unit file. Manual addition of parameter KUBELET_EXTRA_ARGS=--cloud-provider=aws inside /etc/sysconfig/kubelet file solved the issue and labels are being assigned automatically by AWS.
The error states node(s) had taints that the pod didn't tolerate. You must either remove those taints from the nodes using kubectl taint nodes node_name taint- or add tolerations to the objects you create.
Related
I have an ECS cluster where I place a container that runs as a daemon to monitor all other processes. However, I'm seeing this containers being killed by OOM from time to time without leaving a trace. I just happened to spot one of them being killed. This is causing some log duplication but I wonder if there is a way to trace these restarts because when I look on the ECS Cluster events, there are no information about this tasks being restarted by any means.
I know more from kubernetes so I would say an analogy here. When this happens on kubernetes you would see a RESTARTS counter when you get information from all pods (kubectl get pods) is there any way to find this information on AWS ECS tasks? I'm struggling to find on documentation
I identified the tasks, and also I identified the status of each tasks to gain more information, but I'm unable to find any hint that the process was restarted or killed before.
this is a task detail example
- attachments: []
attributes:
- name: ecs.cpu-architecture
value: x86_64
availabilityZone: us-east-2c
clusterArn: arn:aws:ecs:us-west-2:99999999999:cluster/dev
connectivity: CONNECTED
connectivityAt: '2023-01-24T23:03:23.315000-05:00'
containerInstanceArn: arn:aws:ecs:us-east-2:99999999999:container-instance/dev/eb8875fhfghghghfjyjk88c8f96433b8
containers:
- containerArn: arn:aws:ecs:us-east-2:99999999999:container/dev/05d4a402ee274a3ca90a86e46292a63a/e54af51f-2420-47ab-bff6-dcd4f976ad2e
cpu: '500'
healthStatus: HEALTHY
image: public.ecr.aws/datadog/agent:7.36.1
lastStatus: RUNNING
memory: '750'
name: datadog-agent
networkBindings:
- bindIP: 0.0.0.0
containerPort: 8125
hostPort: 8125
protocol: udp
- bindIP: 0.0.0.0
containerPort: 8126
hostPort: 8126
protocol: tcp
networkInterfaces: []
runtimeId: 75559b7327258d69fe61cac2dfe58b12d292bdb7b3a720c457231ee9e3e4190a
taskArn: arn:aws:ecs:us-east-2:99999999999:task/dev/05d4a402ee274a3ca90a86e46292a63a
cpu: '500'
createdAt: '2023-01-24T23:03:22.841000-05:00'
desiredStatus: RUNNING
enableExecuteCommand: false
group: service:datadog-agent
healthStatus: HEALTHY
lastStatus: RUNNING
launchType: EC2
memory: '750'
overrides:
containerOverrides:
- name: datadog-agent
inferenceAcceleratorOverrides: []
pullStartedAt: '2023-01-24T23:03:25.471000-05:00'
pullStoppedAt: '2023-01-24T23:03:39.790000-05:00'
startedAt: '2023-01-24T23:03:47.514000-05:00'
startedBy: ecs-svc/1726924224402147943
tags: []
taskArn: arn:aws:ecs:us-west-2:99999999999:task/dev/05d4a402ee274a3ca90a86e46292a63a
taskDefinitionArn: arn:aws:ecs:us-west-2:99999999999:task-definition/datadog-agent-task:5
version: 2
I don't think ECS tracks or exposes a restart counter for tasks. If you want to be notified of tasks restarting you can create an Event Bridge subscription.
You can use ECS Event with EventBridge and add any action like logging when such event happen.
So, after debugging a lot within the little information AWS provides for this use case, I ended up doing a process to find the answer:
List all tasks ids of a given service with aws-cli with flag --desired-status STOPPED and dump all to a json file
aws ecs list-tasks --cluster dev --service-name datadog-agent
--desired-status STOPPED --output json > ecs_tasks.json
using jq and aws-cli, describe all previously found tasks ids to get further information on each one of them
aws ecs describe-tasks --cluster dev --tasks $(jq -j '.taskArns[] |
(.|" ",.)' ./ecs_tasks.json) --output yaml > ecs_tasks_describe.log
I could came up with a script to group and summarize the information but, since I only had to watch over 20 stopped tasks I ended up dumping the information in yaml format for easiness. I found two key properties on the output:
For each task object, there is a reason for why it was stopped that told me nothing more than it was stopped because a container within the task exited (doesn't say the exit code to help though)
stoppedReason: Essential container in task exited
* For each task object, there is an array of containers objects under **containers** property. There you'll sometimes find **reason** property which can explain a bit more of why the container stopped
reason: 'OutOfMemoryError: Container killed due to memory usage'
Note: This information would give you all events for a given service for at least the last hour. In my case it gave me 8 hours of events but AWS documentation only promises 1 hour https://docs.aws.amazon.com/AmazonECS/latest/developerguide/stopped-task-errors.html
Stopped tasks only appear in the Amazon ECS console, AWS CLI, and AWS SDKs for at least 1 hour after the task stops. After that, the details of the stopped task expire and aren't available in Amazon ECS.
I'm running AWS EKS and trying to integrate it with Datadog.
Kubernetes operates with node names, so
> kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-x-x-x-x.eu-west-1.compute.internal Ready <none> 1d v1.21.1-eks
But all the metrics in Datadog have AWS instance id in host tag, like host:i-012aa34a56a789a0a
And when I see something in the Datadog, I'd like to find the node in kubernetes cluster and investigate deeper inside the node.
For example, to get all the pods in the node
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=ip-x-x-x-x.eu-west-1.compute.internal
But Datadog gives me only host:i-012aa34a56a789a0a.
Is there a way to convert host:i-012aa34a56a789a0a into ip-x-x-x-x.eu-west-1.compute.internal?
Github repo: https://github.com/oussamabouchikhi/udagram-microservices
After I configured the kubectl with the AWS EKS cluster, I deployed the services using these commands
kubectl apply -f env-configmap.yaml
kubectl apply -f env-secret.yaml
kubectl apply -f aws-secret.yaml
# this is repeated for all services
kubectl apply -f svcname-deploymant.yaml
kubectl apply -f svcname-service.yaml
But the pods took hours and still in PENDING state, and when I run the command kubectl describe pod <POD_NAME> I get the follwing info
reverseproxy-667b78569b-2c6hv pod: https://pastebin.com/3xF04SEx
udagram-api-feed-856bbc5c45-jcgtk pod: https://pastebin.com/5UqB79tU
udagram-api-users-6fbd5cbf4f-qbmdd pod: https://pastebin.com/Hiqe1LAM
From your kubectl describe pod <podname>
Warning FailedScheduling 2m19s (x136 over 158m) default-scheduler 0/2 nodes are available: 2 Too many pods.
When you see this, it means that your nodes in AWS EKS is full.
To solve this, you need to add more (or bigger) nodes.
You can also investigate your nodes, e.g. list your nodes with:
kubectl get nodes
and investigate a specific node (check how many pods it has capacity for - and how many pods that runs on the node) with:
kubectl describe node <node-name>
I'm trying to create AWS ALB-Ingress through EKS following the steps in the document https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html
I was successful till the step 7 in creating the controller:
[ec2-user#ip-X-X-X-X eks-cluster]$ kubectl apply -f v2_0_0_full.yaml
customresourcedefinition.apiextensions.k8s.io/targetgroupbindings.elbv2.k8s.aws created
mutatingwebhookconfiguration.admissionregistration.k8s.io/aws-load-balancer-webhook created
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
serviceaccount/aws-load-balancer-controller configured
role.rbac.authorization.k8s.io/aws-load-balancer-controller-leader-election-role created
clusterrole.rbac.authorization.k8s.io/aws-load-balancer-controller-role created
rolebinding.rbac.authorization.k8s.io/aws-load-balancer-controller-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/aws-load-balancer-controller-rolebinding created
service/aws-load-balancer-webhook-service created
deployment.apps/aws-load-balancer-controller created
certificate.cert-manager.io/aws-load-balancer-serving-cert created
issuer.cert-manager.io/aws-load-balancer-selfsigned-issuer created
validatingwebhookconfiguration.admissionregistration.k8s.io/aws-load-balancer-webhook created
However, the controller does NOT get to "Ready" status:
[ec2-user#ip-X-X-X-X eks-cluster]$ kubectl get deployment -n kube-system aws-load-balancer-controller
NAME READY UP-TO-DATE AVAILABLE AGE
aws-load-balancer-controller 0/1 1 0 29m
I'm also able to list the pod associated with the controller which also shows NOT READY:
[ec2-user#ip-X-X-X-X eks-cluster]$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
aws-load-balancer-controller-XXXXXXXXXX-p4l7f 0/1 Pending 0 30m
I also can't seem to get its logs in order to try and debug the issue:
[ec2-user#ip-X-X-X-X eks-cluster]$ kubectl -n kube-system logs aws-load-balancer-controller-XXXXXXXXXX-p4l7f
[ec2-user#ip-X-X-X-X eks-cluster]$
Furthermore, the /var/log directory also does not have any related logs.
Please help me understand why it is not coming to READY state. Also let me know how to enable logging to debug these kind of issues.
I found the answer here. A faragate deployment requires the region and vpc-id.
helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \
--set clusterName=<cluster-name> \
--set serviceAccount.create=false \
--set region=<region-code> \
--set vpcId=<vpc-xxxxxxxx>> \
--set serviceAccount.name=aws-load-balancer-controller \
-n kube-system
From the current LB conntroller manifest I found out that LB controller Pod specification doesn't have Readiness probe, only Liveness probe. That means that the Pod becomes Ready as soon as it pass the Liveness probe:
livenessProbe:
failureThreshold: 2
httpGet:
path: /healthz
port: 61779
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 10
But as we can see in the following output, LB controller's Pod is in Pending state:
[ec2-user#ip-X-X-X-X eks-cluster]$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
aws-load-balancer-controller-XXXXXXXXXX-p4l7f 0/1 Pending 0 30m
If Pod stays in Pending state, it means that kube-scheduler is unable to bind the Pod to a cluster node for whatever reason.
Kube-scheduler is a part of Kubernetes control plain that is responsible for assigning Pods to Nodes.
No Pod logs exist at this phase, because Pod's containers are not started yet.
The most convenient way to check the reason is using the kubectl describe command:
kubectl describe pod/podname -n namespacename
On the bottom of the output there are list of events related to the Pod life cycle. Here is an example for the generic Ubuntu Pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 37s default-scheduler Successfully assigned default/ubuntu to k8s-w1
Normal Pulling 25s (x2 over 35s) kubelet, k8s-w1 Pulling image "ubuntu"
Normal Pulled 23s (x2 over 30s) kubelet, k8s-w1 Successfully pulled image "ubuntu"
Normal Created 23s (x2 over 30s) kubelet, k8s-w1 Created container ubuntu
Normal Started 23s (x2 over 29s) kubelet, k8s-w1 Started container ubuntu
kubectl get events command can also show the problem. For example:
LAST SEEN TYPE REASON OBJECT MESSAGE
21s Normal Scheduled pod/ubuntu Successfully assigned default/ubuntu to k8s-w1
9s Normal Pulling pod/ubuntu Pulling image "ubuntu"
7s Normal Pulled pod/ubuntu Successfully pulled image "ubuntu"
7s Normal Created pod/ubuntu Created container ubuntu
7s Normal Started pod/ubuntu Started container ubuntu
or there could be a reason why Scheduler can't assign Pod to a Node:
"No nodes are available that match all of the predicates: Insufficient cpu (2), Insufficient memory (2)".
In some cases errors could be found in kube-scheduler Pod logs in kube-system namespace. The logs could be listed using the following command:
kubectl logs $(kubectl get pods -l component=kube-scheduler,tier=control-plane -n kube-system -o name) -n kube-system
Most common reasons why pod isn't scheduled are the following:
lack of CPU or memory resources requested by a Pod on the Nodes.
Pod cannot tolerate Taints on the Nodes
Pod have Affinity/AntiAffinity configuration that prevents it from scheduling
Storage or other specific resource (like GPU) requirements in Pod spec cannot be satisfied
I am trying my hands on Kubernetes and I tried to deploy an image into k8s service
root#KubernetesMiniKube:/usr/local/bin# kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.10 --port=8080
pod/hello-minikube created
root#KubernetesMiniKube:/usr/local/bin# kubectl get pod
NAME READY STATUS RESTARTS AGE
hello-minikube 1/1 Running 0 16s
root#KubernetesMiniKube:/usr/local/bin# kubectl get deployments
No resources found in default namespace.
Why i am seeing No resource found but actually there is a resource running inside default namespace.
When you are using $ kubectl run it will create a pod.
In your example thats exactly what happned, it created pod, named hello-minikube.
pod/hello-minikube created
If you want to create deployment
Deployments represent a set of multiple, identical Pods with no unique identities. A Deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive.
you can do it using command:
$ kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.10 --port=8080
deployment.apps/hello-minikube created
user#cloudshell:$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
hello-minikube 1/1 1 1 8s
You can also create deployment using YAML.
Save YAML from this documentation example and use kubectl apply.
$ vi nginx.yaml
<paste proper YAML definition. Also you can use nano editor, or download ready yaml>
user#cloudshell:$ kubectl apply -f nginx.yaml
deployment.apps/nginx-deployment created
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
hello-minikube 1/1 1 1 3m48s
nginx-deployment 3/3 3 3 64s
Please let me know if you have further questions regarding this answer.