Istio 1.6.5 gateway timeout errors - istio

Question

Intermittently we are seeing 504 gateway time out (504) errors when accessing application from browser. We upgraded istio from 1.4.3 to 1.6.5. There was no issue with 1.4.3.

Answer 1

Basically if you want to upgrade istio from 1.4.x to 1.6.x you should first upgrade from 1.4.x to 1.5.x, then upgrade from 1.5.x to 1.6.x
I have followed a theme on istio discuss about upgrade created by#laurentiuspurba.
I have changed it a little for your use case, so an upgrade from 1.4.3 to 1.5.0, then from 1.5.0 to 1.6.8
Take a look at below steps, before using that on your environment I would suggest to test that on some test environment.
1.Follow istio documentation and install istioctl 1.4.3 and 1.5 with:
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.4.3 sh -
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.5.0 sh -
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.6.8 sh -
2.Add the istioctl 1.4.3 to your path
cd istio-1.4.3
export PATH=$PWD/bin:$PATH
3.Install istio 1.4.3
istioctl manifest generate > $HOME/generated-manifest.yaml
kubectl create namespace istio-system
kubectl apply -f generated-manifest.yaml
4.Check if everything works correct.
kubectl get pod -n istio-system
kubectl get svc -n istio-system
istioctl version
5.Add the istioctl 1.5 to your path
cd istio-1.5.0
export PATH=$PWD/bin:$PATH
6.Install istio operator for future upgrade.
istioctl operator init
7.Prepare IstioOperator.yaml
nano IstioOperator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: example-istiocontrolplane
spec:
profile: default
tag: 1.5.0
8.Before the upgrade use below commands
kubectl -n istio-system delete service/istio-galley deployment.apps/istio-galley
kubectl delete validatingwebhookconfiguration.admissionregistration.k8s.io/istio-galley
9.Upgrade from 1.4.3 to 1.5 with istioctl upgrade and prepared IstioOperator.yaml
istioctl upgrade -f IstioOperator.yaml
10.After the upgrade use below commands
kubectl -n istio-system delete deployment istio-citadel istio-galley istio-pilot istio-policy istio-sidecar-injector istio-telemetry
kubectl -n istio-system delete service istio-citadel istio-policy istio-sidecar-injector istio-telemetry
kubectl -n istio-system delete horizontalpodautoscaler.autoscaling/istio-pilot horizontalpodautoscaler.autoscaling/istio-telemetry
kubectl -n istio-system delete pdb istio-citadel istio-galley istio-pilot istio-policy istio-sidecar-injector istio-telemetry
kubectl -n istio-system delete deployment istiocoredns
kubectl -n istio-system delete service istiocoredns
11.Check if everything works correct.
kubectl get pod -n istio-system
kubectl get svc -n istio-system
istioctl version
12.I have deployed a bookinfo app to check if everything work correct.
kubectl label namespace default istio-injection=enabled
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml
13.Results
curl -v xx.xx.xxx.xxx/productpage | grep HTTP
HTTP/1.1 200 OK
istioctl version
client version: 1.5.0
control plane version: 1.5.0
data plane version: 1.5.0 (8 proxies)
14.Add the istioctl 1.6.8 to your path
cd istio-1.6.8
export PATH=$PWD/bin:$PATH
15.Prepare IstioOperator.yaml
nano IstioOperator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: example-istiocontrolplane
spec:
profile: default
tag: 1.6.8
16.Upgrade from 1.5.0 to 1.6.8 with istioctl upgrade and prepared IstioOperator.yaml
istioctl upgrade -f IstioOperator.yaml
17.To upgrade the Istio data plane, you will need to re-inject it.
If you’re using automatic sidecar injection, you can upgrade the sidecar by doing a rolling update for all the pods:
kubectl rollout restart deployment --namespace <namespace with auto injection>
If you’re using manual injection, you can upgrade the sidecar by executing:
kubectl apply -f < (istioctl kube-inject -f <original application deployment yaml>)
18.Results
curl -v xx.xx.xxx.xxx/productpage | grep HTTP
HTTP/1.1 200 OK
istioctl version
client version: 1.6.8
control plane version: 1.6.8
data plane version: 1.6.8 (8 proxies)
Hope you find this useful. If you have any questions let me know.

Answer 2

default timeout = 15 sec.
you can set an explicit timeout in the Virtual Service