Which part of my AWS EKS setup isn't running correctly? - amazon-web-services

Question

I have setup an EKS cluster and configured it to run 6 or so different microservices in their own pods. I am using an ALB as the ingress to said pods and have noticed that sometimes the connection to the pods will time out. I am struggling to determine exactly what the cause of this is.
The pods work as expected for the first X amount of requests but once they have been left for a while and then I try to make a new request, the connection will time out. Could this be due to the ALB or am I missing something with Kubernetes?
One of the deployments looks like this:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: challengepasswordlookupservice
labels:
app: challengepasswordlookupservice
spec:
replicas: 1
selector:
matchLabels:
app: challengepasswordlookupservice
template:
metadata:
labels:
app: challengepasswordlookupservice
spec:
containers:
- name: challengepasswordlookupservice
image: *withheld*
ports:
- containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
name: challengepasswordlookupservice
spec:
type: NodePort
selector:
app: challengepasswordlookupservice
ports:
- protocol: TCP
port: 80
targetPort: 80
And the ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: golf-high-service-ingress
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":{ "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-2:206106816545:certificate/8d91b886-9d57-4fcf-b016-04959cf4d97d
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=60
spec:
rules:
- host: "*withheld*"
http:
paths:
- path: "/*"
pathType: Prefix
backend:
service:
name: ssl-redirect
port:
name: use-annotation
- path: "/*"
pathType: Prefix
backend:
service:
name: challengepasswordlookupservice
port:
number: 80

Answer 1

Try to increase the idle_timeout.timeout_seconds=60 to something relevant like maybe 1800 inside the annotations and then try it out.
I believe you'll want the load-balancer-attributes annotation set to idle_timeout.timeout_seconds=1800
https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/8c2fbfd478a77f983afde2bb5a3ce2f06957e75d/internal/alb/lb/attributes.go#L20

Answer 2

The issue was with the ALB. On AWS, an ALB requires 1 subnet in at least two different availability zones. One of the subnets that was attached did not have an internet gateway attached and thus was not establishing a connection when the request was routed to this subnet.