Issue pulling secret for NGINX Ingress Controller with NLB in EKS - amazon-web-services

Question

Trying to provision NGINX ingress controller with NLB in EKS. Getting CrashLoopBackOff for ingress-nginx-admission-create and ingress-nginx-admission-patch.
It's worth mentioning that this is a private EKS cluster without internet access and I'm pulling the docker images successfully from ECR. Also, I am using a secondary VPC CIDR to allocate pod IPs.
I am following this documentation:
https://kubernetes.github.io/ingress-nginx/deploy/#aws
https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/aws/deploy.yaml
The following resources are created:
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-admission-create-lws8l 0/1 CrashLoopBackOff 4 4m18s
ingress-nginx ingress-nginx-admission-patch-g5s8w 0/1 CrashLoopBackOff 4 4m18s
ingress-nginx ingress-nginx-controller-79c469cd9f-wqmhn 0/1 ContainerCreating 0 4m18s
kube-system aws-node-4g2h2 1/1 Running 0 29h
kube-system aws-node-r65xb 1/1 Running 0 29h
kube-system aws-node-spfzj 1/1 Running 0 29h
kube-system coredns-65ccb76b7c-97xcv 1/1 Running 0 31h
kube-system coredns-65ccb76b7c-bck8f 1/1 Running 0 31h
kube-system kube-proxy-jnfjd 1/1 Running 0 29h
kube-system kube-proxy-smc88 1/1 Running 0 29h
kube-system kube-proxy-v6hjp 1/1 Running 0 29h
One of the issues seems to be getting the secret and I don't understand why.
{"err":"Get \"https://172.20.0.1:443/api/v1/namespaces/ingress-nginx/secrets/ingress-nginx-admission\": dial tcp 172.20.0.1:443: i/o timeout","level":"fatal","msg":"error getting secret","source":"k8s/k8s.go:232","time":"2022-03-11T23:56:16Z"}
The other thing is that the NLB is not being created.
sh-4.2$ kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 172.20.76.158 <pending> 80:31054/TCP,443:32308/TCP 7m43s
ingress-nginx-controller-admission ClusterIP 172.20.240.214 <none> 443/TCP 7m43s
sh-4.2$ kubectl get -A ValidatingWebhookConfiguration
NAME WEBHOOKS AGE
ingress-nginx-admission 1 12m
vpc-resource-validating-webhook 1 31h
Here is the ingress-nginx-admission-create-xw4rz pod.
Name: ingress-nginx-admission-create-xw4rz
Namespace: ingress-nginx
Priority: 0
Node: ip-10-51-80-103.eu-west-2.compute.internal/10.51.80.103
Start Time: Sat, 12 Mar 2022 14:38:38 +0000
Labels: app.kubernetes.io/component=admission-webhook
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=1.1.1
controller-uid=e6d21979-55bf-4b44-aa41-6462b9923806
helm.sh/chart=ingress-nginx-4.0.15
job-name=ingress-nginx-admission-create
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.0.25.33
IPs:
IP: 10.0.25.33
Controlled By: Job/ingress-nginx-admission-create
Containers:
create:
Container ID: docker://9a311111111111111111111111111111111111111111111111111 Image: 11111111111111.dkr.ecr.eu-west-2.amazonaws.com/certgen:latest
Image ID: docker-pullable://11111111111111.dkr.ecr.eu-west-2.amazonaws.com/certgen#sha256:7831111111111111111111111111111
Port: <none>
Host Port: <none>
Args:
create
--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
--namespace=$(POD_NAMESPACE)
--secret-name=ingress-nginx-admission
State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 12 Mar 2022 14:42:05 +0000
Finished: Sat, 12 Mar 2022 14:42:35 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 12 Mar 2022 14:40:52 +0000
Finished: Sat, 12 Mar 2022 14:41:22 +0000
Ready: False
Restart Count: 4
Environment:
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rrlnd (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-rrlnd:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m59s default-scheduler Successfully assigned ingress-nginx/ingress-nginx-admission-create-xw4rz to ip-10-51-80-103.eu-west-2.compute.internal
Normal Pulled 32s (x5 over 3m58s) kubelet Container image "111111111111111.dkr.ecr.eu-west-2.amazonaws.com/certgen:latest" already present on machine
Normal Created 32s (x5 over 3m58s) kubelet Created container create
Normal Started 32s (x5 over 3m58s) kubelet Started container create
Warning BackOff 2s (x7 over 2m56s) kubelet Back-off restarting failed container
And here is the ingress-nginx-controller-79c469cd9f-ft76q. I can see a failed mount, but I understand that "ingress-nginx-controller" is created after the 2 pods above run without errors.
sh-4.2$ kubectl describe pod ingress-nginx-controller-79c469cd9f-ft76q -n ingress-nginx
Name: ingress-nginx-controller-79c469cd9f-ft76q
Namespace: ingress-nginx
Priority: 0
Node: ip-10-51-80-6.eu-west-2.compute.internal/10.51.80.6
Start Time: Sat, 12 Mar 2022 14:38:38 +0000
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=79c469cd9f
Annotations: kubernetes.io/psp: eks.privileged
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/ingress-nginx-controller-79c469cd9f
Containers:
controller:
Container ID:
Image: 111111111111111.dkr.ecr.eu-west-2.amazonaws.com/nginx-controller:latest
Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
--election-id=ingress-controller-leader
--controller-class=k8s.io/ingress-nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-79c469cd9f-ft76q (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j8v4c (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-j8v4c:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m46s default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-79c469cd9f-ft76q to ip-10-51-80-6.eu-west-2.compute.internal
Warning FailedMount 6m43s kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[kube-api-access-j8v4c webhook-cert]: timed out waiting for the condition
Warning FailedMount 2m14s (x2 over 4m29s) kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert kube-api-access-j8v4c]: timed out waiting for the condition
Warning FailedMount 32s (x12 over 8m46s) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found