Istio retry not works - istio

Question

There are two service, ws001-api invokes ws002-pingpong via gRPC
######## ws001-api #########
apiVersion: apps/v1
kind: Deployment
metadata:
name: ws001-api
labels:
app: ws001-api
spec:
replicas: 1
selector:
matchLabels:
app: ws001-api
strategy:
type: RollingUpdate
template:
metadata:
labels:
auth-injection: enabled
app: ws001-api
spec:
containers:
- name: ws001-api
image: yw4code/ws001-api:latest
imagePullPolicy: Always
ports:
- name: http
containerPort: 7001
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: ws001-api
spec:
type: NodePort
selector:
app: ws001-api
ports:
- name: http
protocol: TCP
port: 7001
targetPort: 7001
######## ws002-pingpong #########
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ws002-pingpong
labels:
app: ws002-pingpong
spec:
replicas: 2
selector:
matchLabels:
app: ws002-pingpong
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
labels:
app: ws002-pingpong
spec:
containers:
- name: ws002-pingpong
image: yw4code/ws002-pingpong:latest
imagePullPolicy: Always
ports:
- name: grpc
containerPort: 7002
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: ws002-pingpong
spec:
type: NodePort
selector:
app: ws002-pingpong
ports:
- name: grpc
protocol: TCP
port: 7002
targetPort: 7002
---
And here is my retry definition
kind: VirtualService
metadata:
name: retry
spec:
hosts:
- ws002-pingpong
http:
- timeout: 3s
route:
- destination:
host: ws002-pingpong
port:
number: 7002
retries:
attempts: 3
perTryTimeout: 1s
retryOn: unavailable,cancelled,deadline-exceeded,internal,resource-exhausted,connect-failure,gateway-error,refused-stream
I added an panic keep ws002-pinpong (container) getting crash, made k8s retry it
ReplicaSet "ws002-pingpong-d459bb59b" has timed out progressing.; Deployment does not have minimum availability.
istio-proxy in ws002-pingpong pod is alive. but didn't get any message.
ws001-api get following response via gRPC
rpc error: code = Unavailable desc = no healthy upstream
But still can't get any metrics about retrying in prometheus.
metric envoy_cluster_upstream_rq_retry of the pod is zero. Can any one figure it out?