Override Istio retry back off interval - istio

I am trying to override Istio's default retry back off interval with the help of an EnvoyFilter.
I have three services, each calling it's successor. Service-2 has retries enabled with a VirtualService.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
labels:
app: service-2
name: service-2-vs
namespace: retry-test
spec:
hosts:
- service-2
http:
- route:
- destination:
host: service-2
retries:
attempts: 5
retryOn: 5xx
The retries are working, but when I apply an EnvoyFilter to override the default retry back off interval I see no effects.
I used the following EnvoyFilter for overriding the back off intervals.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: service-2-envoy-config
namespace: retry-test
spec:
workloadSelector:
labels:
app: service-2
configPatches:
- applyTo: HTTP_ROUTE
match:
context: SIDECAR_OUTBOUND
routeConfiguration:
vhost:
name: "service-2.retry-test.svc.cluster.local:5002"
patch:
operation: MERGE
value:
route:
retry_policy:
retry_back_off:
base_interval: "1s"
max_interval: "5s"
I also tried configuring the EnvoyFilter for Service-1 since this would be the service sending requests to Service-2, but this didn't work either.
When checking with Kiali I can see that the EnvoyFilter gets applied to the correct service and when looking at Envoy configs of the workload I can see the following got applied.
"route": {
"cluster": "outbound|5002||service-2.retry-test.svc.cluster.local",
"max_grpc_timeout": "0s",
"retry_policy": {
"host_selection_retry_max_attempts": "5",
"num_retries": 5,
"retry_back_off": {
"base_interval": "1s",
"max_interval": "5s"
},
...
}
}
Can someone help me to figure out how to apply the right EnvoyFilter to override the default back off interval?

Posting this if anyone runs into the same problem and finds this question.
When using an EnvoyFilter you need to apply the filter for the service that will send the requests with the vhost being the service the requests are sent to.
In this case this would mean the Envoyfilter gets applied to service-1 with vhost of the routeConfiguration being service-2.
The according yaml file looks like this:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: service-1-envoy-config
namespace: retry-test
spec:
workloadSelector:
labels:
app: service-1
configPatches:
- applyTo: HTTP_ROUTE
match:
context: SIDECAR_OUTBOUND
routeConfiguration:
vhost:
name: "service-2.retry-test.svc.cluster.local:5002"
patch:
operation: MERGE
value:
route:
retry_policy:
retry_back_off:
base_interval: "1s"
max_interval: "5s"

Related

Istio VirtualService delegate with mesh gateway

I am implementing some routing logic for a set of internal services where a delegate VirtualService looks like a good solution:
https://istio.io/latest/docs/reference/config/networking/virtual-service/#Delegate
I created some test setup similar to the one in the documentation with only one difference. In my case, the "root" VirtualService bound to the "mesh" Gateway and the "host" is then some internal service name. Is this supposed to work or does delegation only work with non-mesh Gateways?
This is the root VirtualService (the idea is that all requests are sent to worker-pool.default.svc.cluster.local and depending on some HTTP headers they are then forwarded to other VirtualServices):
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: worker-pool
spec:
hosts:
- worker-pool.default.svc.cluster.local
http:
- name: "route 1"
match:
- headers:
customer-id:
exact: alice
delegate:
name: worker-for-alice
- name: "route 2"
match:
- headers:
customer-id:
exact: bob
delegate:
name: worker-for-bob
And here the other VirtualService (only showing one, both look the same):
apiVersion: v1
kind: Service
metadata:
name: worker-for-alice
labels:
app: worker-for-alice
service: worker-for-alice
spec:
...
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: worker-for-alice
spec:
http:
- route:
- destination:
host: worker-for-alice
I guess there are some points to consider:
As far as I am concerned, Delegate will only work for Gateways. So, you should need something like this:
spec:
gateways:
- mesh
hosts:
- worker-pool.default.svc.cluster.local
http:
...
It is always a good idea to use namespaces. Once you have the items defined in namespaces, add "namespace" element to delegate definition:
delegate:
name: worker-for-bob
namespace: <some-namespace>
Last, but not least, it may be necessary to setup variable PILOT_ENABLE_VIRTUAL_SERVICE_DELEGATE to "true" on istiod configuration:
kubectl edit deployment istiod -n istio-system
And then add this env variable in spec.template.spec.container.env

Set request buffer for Istio EnvoyFilter in YAML

Can you help me please to specify YAML Istio EnvoyFilter to have the request buffer, analog of Nginx request buffer.
I tried
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: connection
namespace: my-test
spec:
workloadSelector:
labels:
role: backend
configPatches:
- applyTo: LISTENER
match:
context: SIDECAR_INBOUND
patch:
operation: MERGE
value:
per_connection_buffer_limit_bytes: 21000000.0 #21MB
This is applied successfully to the cluster but does not work as expected
Thanks
UPD
this works (for istio 1.5)
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: connection
namespace: my-test
spec:
filters:
- filterName: envoy.buffer
filterType: HTTP
filterConfig:
maxRequestBytes: 21000000.0 #21MB
UPD 2
this works (for istio 1.8)
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: filter-buffersize-limit
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
subFilter:
name: "envoy.filters.http.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.buffer
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.buffer.v3.Buffer
max_request_bytes: 26214400 # 25 MB.

Configuring EnvoyFilter in Istio for "set_current_client_cert_details"

I have a scenario where I am terminating the TLS traffic at my Istio Ingress Gateway and forwarding it to respective services. Also have mTls and ISTIO_MUTUAL set. I need to validate the client certificate that is being sent when calling our Ingress-Gateway in the cluster and I am planning to do it in code for which I need the entire cert forwarded to my service code.
I tried configuring the EnvoyFilter as below :
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: xfcc-forward
namespace: xfcc
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
sni: "*"
patch:
operation: MERGE
value:
set_current_client_cert_details:
cert: true
This fails while applying and I guess it might be because I am not configuring it right. Could someone help me with the right configuration of the EF for my scenario.
I think I got the Configuration working with this :
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: xfcc-forward
namespace: xfcc
spec:
configPatches:
- applyTo: NETWORK_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
patch:
operation: MERGE
value:
typed_config:
"#type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
forward_client_cert_details: ALWAYS_FORWARD_ONLY
set_current_client_cert_details:
subject: true
cert: true
chain: true
I can see the XFCC header with the cert values, but the certs are the ones passed on by Citadel for mTls and not the one that the client has sent in the header, which I am still looking at.

How to make the ext_authz envoy filter work on the istio cluster?

I am trying to add ext_authz filter to istio ingress-gateway for requests authentication. But when I'm adding this filter to cluster it seems like it is not added to envoy configuration, i.e. it is not working.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: authn-filter
namespace: istio-system
spec:
filters:
- insertPosition:
index: FIRST
listenerMatch:
portNumber: 433
listenerType: GATEWAY
listenerProtocol: HTTP
filterType: HTTP
filterName: "envoy.ext_authz"
filterConfig:
http_service:
server-uri:
uri: http://auth.default.svc.cluster.local:8080
cluster: outbound|8080||auth.default.svc.cluster.local
timeout: 2s
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: auth-virtualservice
spec:
hosts:
- "*"
gateways:
- gateway.default.svc.cluster.local
http:
- match:
- uri:
prefix: "/auth"
route:
- destination:
host: auth.default.svc.cluster.local
I figured that out, the problem was caused by the old version of istio deployed on the cluster

create VirtualService for kiali, tracing, grafana

I am trying to expose kiali on my default gateway. I have other services working for apps in the default namespace but have not been able to route traffic to anything in the istio namespace
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- '*'
tls:
httpsRedirect: true
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- '*'
tls:
mode: SIMPLE
privateKey: /etc/istio/ingressgateway-certs/tls.key
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: kiali
namespace: default
spec:
hosts:
- kiali.dev.example.com
gateways:
- gateway
http:
- route:
- destination:
host: kiali.istio-system.svc.cluster.local
port:
number: 20001
The problem was I had mTLS enabled and kiali does not have a sidecar thus can not be validated by mTLS. the solution was to add a destination rule disabling mTLS for it.
apiVersion: 'networking.istio.io/v1alpha3'
kind: DestinationRule
metadata:
name: kiali
namespace: istio-system
spec:
host: kiali.istio-system.svc.cluster.local
trafficPolicy:
tls:
mode: DISABLE
You should define an ingress gateway and make sure that the hosts in the gateway match the hosts in the virtual service. Also specify the port of the destination. See the Control Ingress Traffic task.
For me this worked!
I ran
istioctl proxy-config routes istio-ingressgateway-866d7949c6-68tt4 -n istio-system -o json > ./routes.json
to get the dump of all the routes. The kiali route got corrupted for some reason. I deleted the virtual service and created it again, that fixed it.
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: kiali
namespace: istio-system
spec:
gateways:
- istio-system/my-gateway
hosts:
- 'mydomain.com'
http:
- match:
- uri:
prefix: /kiali/
route:
- destination:
host: kiali.istio-system.svc.cluster.local
port:
number: 20001
weight: 100
---
apiVersion: 'networking.istio.io/v1alpha3'
kind: DestinationRule
metadata:
name: kiali
namespace: istio-system
spec:
host: kiali.istio-system.svc.cluster.local
trafficPolicy:
tls:
mode: SIMPLE
---
Note: hosts needed to be set, '*' didnt work for some reason.