pod stuck on `ContainerCreating` state in AWS EKS - amazon-web-services

Question

I deployed a k8s cluster on AWS EKS fargate. And deployed a elasticsearch container to the pod. The pod is stuck on ContainerCreating state and describe pod shows below error:
$ kubectl describe pod es-0
Name: es-0
Namespace: default
Priority: 2000001000
Priority Class Name: system-node-critical
Node: fargate-ip-10-0-1-207.ap-southeast-2.compute.internal/10.0.1.207
Start Time: Fri, 28 May 2021 16:39:07 +1000
Labels: controller-revision-hash=es-86f54d94fb
eks.amazonaws.com/fargate-profile=elk_profile
name=es
statefulset.kubernetes.io/pod-name=es-0
Annotations: CapacityProvisioned: 1vCPU 2GB
Logging: LoggingDisabled: LOGGING_CONFIGMAP_NOT_FOUND
kubernetes.io/psp: eks.privileged
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/es
Containers:
es:
Container ID:
Image: elasticsearch:7.10.1
Image ID:
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 8
Requests:
cpu: 1
memory: 4
Environment: <none>
Mounts:
/usr/share/elasticsearch/config/elasticsearch.yml from es-config (rw,path="elasticsearch.yml")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-6qql4 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
es-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: es-config
Optional: false
default-token-6qql4:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-6qql4
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 75s (x4252 over 16h) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused \"read init-p: connection reset by peer\"": unknown
How do I know what the issue is and how to fix it? I have tried to restart the Statefulset but it didn't restart. It seems the pod stucked.
apiVersion: v1
kind: ConfigMap
metadata:
name: es-config
data:
elasticsearch.yml: |
cluster.name: my-elastic-cluster
network.host: "0.0.0.0"
bootstrap.memory_lock: false
discovery.zen.ping.unicast.hosts: elasticsearch-cluster
discovery.zen.minimum_master_nodes: 1
discovery.type: single-node
ES_JAVA_OPTS: -Xms2g -Xmx4g
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es
namespace: default
spec:
serviceName: es-entrypoint
replicas: 1
selector:
matchLabels:
name: es
template:
metadata:
labels:
name: es
spec:
volumes:
- name: es-config
configMap:
name: es-config
items:
- key: elasticsearch.yml
path: elasticsearch.yml
# - name: persistent-storage
# persistentVolumeClaim:
# claimName: efs-es-claim
securityContext:
fsGroup: 1000
runAsUser: 1000
runAsGroup: 1000
containers:
- name: es
image: elasticsearch:7.10.1
resources:
limits:
cpu: 2
memory: 8
requests:
cpu: 1
memory: 4
ports:
- name: http
containerPort: 9200
- containerPort: 9300
name: inter-node
volumeMounts:
- name: es-config
mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
subPath: elasticsearch.yml
# - name: persistent-storage
# mountPath: /usr/share/elasticsearch/data
---
apiVersion: v1
kind: Service
metadata:
name: es-entrypoint
spec:
selector:
name: es
ports:
- port: 9200
targetPort: 9200
protocol: TCP
type: NodePort

Answer 1

Figured out why it happens, after remove the limits resources, it works. Not sure why it doesn't allow limits
limits:
cpu: 2
memory: 8