ISTIO External Auth : '503 upstream connect error or disconnect/reset before headers. reset reason: connection terminationroot' when access over HTTPS - istio

Question

ISTIO version: 1.9.4
EKS Cluster version: 1.14
We have deployed ISTIO APP mesh in our project. We have deployed External Authorization using istio's documentation i.e. https://istio.io/latest/docs/tasks/security/authorization/authz-custom/.
External authorizer used (as mentioned in above documentation) : https://raw.githubusercontent.com/istio/istio/release-1.9/samples/extauthz/ext-authz.yaml
When we access any API from going into pod of another API (i.e. over http), using curl command, all works fine. External auth service gets call and all the headers are passed into external authorizer's v3 check method. Below information is passed
source, principal, destination, headers: authority, method, path, accept, content-length, user-agent, x-b3-sampled, x-b3-spanid, x-b3-traceid, x-envoy-attempt-count, x-ext-authz, x-forwarded-client-certx-forwarded-proto, x-request-id.
But when we try to access the same service over https using postman, browser or from going into pod of another API and using curl with https endpoint, we get denied response from external authorizer's v3 check method. Also when we check the logs of external authorizer's v3 check method no headers are passed to it in this case.
Below is setup
Name spaces with ISTIO ejection enable : foo
1. ISTIO Config map changes
data:
mesh: |-
# Add the following content to define the external authorizers.
extensionProviders:
- name: "sample-ext-authz-grpc"
envoyExtAuthzGrpc:
service: "ext-authz.foo.svc.cluster.local"
port: "9000"
- name: "sample-ext-authz-http"
envoyExtAuthzHttp:
service: "ext-authz.foo.svc.cluster.local"
port: "8000"
includeHeadersInCheck: ["x-ext-authz"]
2. External Authorizer
apiVersion: v1
kind: Service
metadata:
name: ext-authz
namespace: foo
labels:
app: ext-authz
spec:
ports:
- name: http
port: 8000
targetPort: 8000
- name: grpc
port: 9000
targetPort: 9000
selector:
app: ext-authz
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ext-authz
namespace: foo
spec:
replicas: 1
selector:
matchLabels:
app: ext-authz
template:
metadata:
labels:
app: ext-authz
spec:
containers:
- image: docker.io/istio/ext-authz:0.6
imagePullPolicy: IfNotPresent
name: ext-authz
ports:
- containerPort: 8000
- containerPort: 9000
3. Enable the external authorization Config
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ext-authz
namespace: foo
spec:
selector:
matchLabels:
app: user-api
action: CUSTOM
provider:
name: sample-ext-authz-grpc
rules:
- to:
- operation:
paths: ["/user/api/*"]
4. PeerAuth Chagnes
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: mtlsauth
namespace: foo
spec:
mtls:
mode: STRICT
5. Destination Rule
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: default
namespace: foo
spec:
host: "*.samplehost.svc.cluster.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
6. Virtual Service File
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: sample-gateway
namespace: foo
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "sample.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: user-api
namespace: foo
spec:
hosts:
- "sample.com"
gateways:
- sample-gateway
http:
- match:
- uri:
prefix: /user/api/
route:
- destination:
host: user-api
port:
number: 9500
Logs from ingress gateway:
2021-07-08T11:13:33.554104Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp 172.20.0.51:15012: connect: connection refused"
2021-07-08T11:13:35.420052Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-07-08T11:43:24.012961Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0

Answer 1

I am not sure if you are facing the issue but if seems like you have enforced mtls . Thats why in the following config for gateway. You might need to open HTTPS also
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: sample-gateway
namespace: foo
spec:
selector:
istio: ingressgateway
servers:
port:
number: 80
name: http
protocol: HTTP
hosts:
"sample.com"
port:
number: 443
name: https
protocol: HTTPS
hosts:
"sample.com"