Kubernetes nginx ingress periodically gives 404 - amazon-web-services

Question

I have deployed kops k8s in AWS, everything in the same namespace.
nginx ingress controller route traffic to https backends (wordpress apps).
I'm able to reach the website, but unfortunately for every 10~ calls only 1 call get http 200. all the other 9 get 404 nginx not found.
tried to search everywhere but no luck :(
My configuration:
DNS -> AWS NLB -> 2 Nodes
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-nginx
namespace: example-ns
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/ssl-passthrough: "True"
nginx.org/ssl-services: test-service
nginx.ingress.kubernetes.io/affinity: "cookie"
spec:
rules:
- host: "test.example.com"
http:
paths:
- path: /
backend:
serviceName: test-service
servicePort: 8443
nginx-service.yaml:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: example-ns
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
nginx-daemonset.yaml:
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
name: nginx-ingress-controller
namespace: example-ns
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
serviceAccountName: nginx-ingress-serviceaccount
imagePullSecrets:
- name: private-repo
containers:
- name: nginx-ingress-controller
image: private_repo/private_image
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
- --default-ssl-certificate=$(POD_NAMESPACE)/tls-cert
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
runAsUser: 33
resources:
limits:
cpu: 500m
memory: 300Mi
requests:
cpu: 400m
memory: 200Mi
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
wordpress.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-example
namespace: example-ns
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
strategy:
type: RollingUpdate
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
restartPolicy: Always
volumes:
- name: volume
persistentVolumeClaim:
claimName: volume-claim
imagePullSecrets:
- name: private-repo
containers:
- name: test-example-httpd
image: private_repo/private_image
imagePullPolicy: Always
ports:
- containerPort: 8443
name: https
- name: test-example-php-fpm
image: private_repo/private_image
imagePullPolicy: Always
securityContext:
runAsUser: 82
securityContext:
allowPrivilegeEscalation: false
---
apiVersion: v1
kind: Service
metadata:
name: test-service
namespace: example-ns
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
ports:
- name: https-web
targetPort: 8443
port: 8443
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---UPDATE---
kubectl get endpoints,services -n example-ns
NAME ENDPOINTS AGE
endpoints/ingress-nginx 100.101.0.1:8443,100.100.0.4:443,100.101.0.2:443 1d
endpoints/test-service 100.100.0.1:8443,100.101.0.1:8443,100.101.0.2:8443 4h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx LoadBalancer SOME-IP sometext.elb.us-west-3.amazonaws.com 80:31541/TCP,443:31017/TCP 1d
service/test-service ClusterIP SOME-IP <none> 8443/TCP 4h
Thanks!

Answer 1

Apparently changing the annotation nginx.ingress.kubernetes.io/ssl-passthrough from "True" to "False" solved it.
Probably has to do something with ssl termination in NGINX and not in the apache.