How to gzip stats endpoint in Istio? - istio

Question

We are scraping the metrics of many istio-proxy sidecars with Prometheus. As these are many metrics, we would like to compress the payload to save us some bandwidth.
Out of the box the stats endpoint does not seem to be compressed with Istio 1.8.2:
$ kubectl exec -it my-pod-0 -c server -- curl -o /dev/null -vsS --compressed http://127.0.0.1:15090/stats/prometheus
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 15090 (#0)
> GET /stats/prometheus HTTP/1.1
> Host: 127.0.0.1:15090
> User-Agent: curl/7.61.1
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
< content-type: text/plain; charset=UTF-8
< cache-control: no-cache, max-age=0
< x-content-type-options: nosniff
< date: Fri, 19 Feb 2021 10:42:25 GMT
< server: envoy
< x-envoy-upstream-service-time: 2
< transfer-encoding: chunked
<
{ [26267 bytes data]
* Connection #0 to host 127.0.0.1 left intact
How do I get the sidecar to compress the stats traffic?
So far I tried adding an EnvoyFilter, but I honestly have no idea about the Envoy internals and I failed to find docs that help me understanding it.
My understanding is that I have to add the compress filter to this:
$ istioctl proxy-config listeners maintenance-0 --port 15090 -o json | gron
json = [];
json[0] = {};
json[0].address = {};
json[0].address.socketAddress = {};
json[0].address.socketAddress.address = "0.0.0.0";
json[0].address.socketAddress.portValue = 15090;
json[0].filterChains = [];
json[0].filterChains[0] = {};
json[0].filterChains[0].filters = [];
json[0].filterChains[0].filters[0] = {};
json[0].filterChains[0].filters[0].name = "envoy.filters.network.http_connection_manager";
json[0].filterChains[0].filters[0].typedConfig = {};
json[0].filterChains[0].filters[0].typedConfig.httpFilters = [];
json[0].filterChains[0].filters[0].typedConfig.httpFilters[0] = {};
json[0].filterChains[0].filters[0].typedConfig.httpFilters[0].name = "envoy.filters.http.router";
json[0].filterChains[0].filters[0].typedConfig.httpFilters[0].typedConfig = {};
json[0].filterChains[0].filters[0].typedConfig.httpFilters[0].typedConfig["#type"] = "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router";
json[0].filterChains[0].filters[0].typedConfig.routeConfig = {};
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts = [];
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0] = {};
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].domains = [];
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].domains[0] = "*";
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].name = "backend";
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes = [];
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes[0] = {};
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes[0].match = {};
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes[0].match.prefix = "/stats/prometheus";
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes[0].route = {};
json[0].filterChains[0].filters[0].typedConfig.routeConfig.virtualHosts[0].routes[0].route.cluster = "prometheus_stats";
json[0].filterChains[0].filters[0].typedConfig.statPrefix = "stats";
json[0].filterChains[0].filters[0].typedConfig["#type"] = "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager";
So far I tried creating the filter a few times and this is my latest try:
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: gzip
spec:
workloadSelector:
labels:
app: my-pod
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_INBOUND
listener:
filterChain:
filter:
name: envoy.http_connection_manager
subFilter:
name: envoy.router
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.compressor
typed_config:
'#type': type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
compressor_library:
name: text_optimized
typed_config:
'#type': type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
remove_accept_encoding_header: true
I do not really know what to put into the .spec.configPatches.match section. The patch and applyTo section probably are wrong too.

Answer 1

With help in an Istio issue, we made it work. I am copying my original response from: https://github.com/istio/istio/issues/30987#issuecomment-822517456
I got a working example and our network usage went down from ~20MBytes/s to ~30KBytes/s (yes, from Mega to Kilo 🔥). First I thought there was any error, but the data was complete and I did a short check with my CLI:
$ kubectl exec elasticsearch-0 -c istio-proxy -- timeout 1 curl -Ss --fail --compressed -w '%{size_download}' -i http://localhost:14090/stats/prometheus | tail -n 1
7763
$ kubectl exec elasticsearch-0 -c istio-proxy -- timeout 1 curl -Ss --fail -w '%{size_download}' -i http://localhost:14090/stats/prometheus | tail -n 1
330315
It is only 2.35% of its original size and both have the same mount of lines!
Here is the custom bootstrap, that need to be added to each pod with the sidecar.istio.io/bootstrapOverride: "istio-custom-bootstrap-config"
annotation.
apiVersion: v1
kind: ConfigMap
metadata:
annotations:
name: istio-custom-bootstrap-config
namespace: default
data:
custom_bootstrap.json: |-
{
"staticResources": {
"listeners": [
{
"address": {
"socketAddress": {
"address": "0.0.0.0",
"portValue": 14090
}
},
"filterChains": [
{
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typedConfig": {
"#type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"httpFilters": [
{
"name": "envoy.filters.http.compressor",
"typed_config": {
"#type": "type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor",
"compressor_library": {
"name": "text_optimized",
"typed_config": {
"#type": "type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip"
}
},
"remove_accept_encoding_header": true
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"#type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
}
}
],
"routeConfig": {
"virtualHosts": [
{
"domains": [
"*"
],
"name": "backend",
"routes": [
{
"match": {
"prefix": "/stats/prometheus"
},
"route": {
"cluster": "prometheus_stats"
}
}
]
}
]
},
"statPrefix": "stats"
}
}
]
}
]
}
]
}
}
I had to change the port, because it is easy to maintain when there gets anything added to staticResources.listeners in future updates.