What "max_connections" really mean in Envoy? - istio

Question

My question is about what "cluster.CircuitBreakers.Thresholds.max_connections" really mean in Envoy.
cluster.CircuitBreakers.Thresholds.max_connections in envoy doc is explained as :
The maximum number of connections that Envoy will make to the upstream cluster. If not specified, the default is 1024.
Istio uses envoy as a sidecar. Recently we tried the circuit breaking sample but always found that there are more connections than we configured.
So we make another test shown below:
Add two services into istio:
echo client: 1 pod, downstream, will send HTTP request to echo server
echo server: 2 pods, upstream.
the service pods:
[root#k8s-master istio-1.0.3]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
echoclient-84485fbc5c-zxlv8 2/2 Running 0 8s 10.244.2.79 node02 <none>
echoserver-5655768fb9-smsvb 2/2 Running 0 23h 10.244.2.65 node02 <none>
echoserver-5655768fb9-srsq2 2/2 Running 0 7h52m 10.244.2.73 node02 <none>
Configured destination rule for echo server, below shows the corresponding info in the envoy. (maxConnections is 2)
istio proxy-config output:
[root#k8s-master istio-1.0.3]# istioctl proxy-config clusters echoclient-84485fbc5c-zxlv8 --fqdn echoserver.default.svc.cluster.local -o json
[
{
"name": "outbound|8080||echoserver.default.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {}
},
"serviceName": "outbound|8080||echoserver.default.svc.cluster.local"
},
"connectTimeout": "1.000s",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2
}
]
}
}
]
Make multiple requests concurrently(40 requests per thread) from echo client to each server.
results:
[root#k8s-master istio-1.0.3]# kubectl exec -it echoclient-84485fbc5c-zxlv8 /bin/bash
Defaulting container name to echoclient.
Use 'kubectl describe pod/echoclient-84485fbc5c-zxlv8 -n default' to see all of the containers in this pod.
[root#echoclient-84485fbc5c-zxlv8 /]# /opt/jre/bin/java -cp /opt/echoclient-1.0-SNAPSHOT-jar-with-dependencies.jar hello.HttpSender "http://echoserver:8080/echo?name=peter" 10 40 0
using num threads: 10
Starting pool-1-thread-1 with numCalls=40 parallelSends=false delayBetweenCalls=0 url=http://echoserver:8080/echo?name=peter mixedRespTimes=false
Starting pool-1-thread-2 with numCalls=40 parallelSends=false delayBetweenCalls=0 url=http://echoserver:8080/echo?name=peter mixedRespTimes=false
Starting pool-1-thread-3 with numCalls=40 parallelSends=false delayBetweenCalls=0 url=http://echoserver:8080/echo?name=peter mixedRespTimes=false
Starting pool-1-thread-4 with numCalls=40 parallelSends=false delayBetweenCalls=0 url=http://echoserver:8080/echo?name=peter mixedRespTimes=false
Starting pool-1-thread-5 with numCalls=40 parallelSends=false delayBetweenCalls=0 url=http://echoserver:8080/echo?name=peter mixedRespTimes=false
Starting pool-1-thread-6 with numCalls=40 parallelSends=false delayBetweenCalls=0 url=http://echoserver:8080/echo?name=peter mixedRespTimes=false
Starting pool-1-thread-7 with numCalls=40 parallelSends=false delayBetweenCalls=0 url=http://echoserver:8080/echo?name=peter mixedRespTimes=false
Starting pool-1-thread-8 with numCalls=40 parallelSends=false delayBetweenCalls=0 url=http://echoserver:8080/echo?name=peter mixedRespTimes=false
Starting pool-1-thread-9 with numCalls=40 parallelSends=false delayBetweenCalls=0 url=http://echoserver:8080/echo?name=peter mixedRespTimes=false
Starting pool-1-thread-10 with numCalls=40 parallelSends=false delayBetweenCalls=0 url=http://echoserver:8080/echo?name=peter mixedRespTimes=false
pool-1-thread-7: successes=[40], failures=[0], duration=[481ms]
pool-1-thread-6: successes=[40], failures=[0], duration=[485ms]
pool-1-thread-4: successes=[40], failures=[0], duration=[504ms]
pool-1-thread-1: successes=[40], failures=[0], duration=[542ms]
pool-1-thread-9: successes=[40], failures=[0], duration=[626ms]
pool-1-thread-8: successes=[40], failures=[0], duration=[652ms]
pool-1-thread-2: successes=[40], failures=[0], duration=[684ms]
pool-1-thread-10: successes=[40], failures=[0], duration=[657ms]
pool-1-thread-5: successes=[40], failures=[0], duration=[678ms]
pool-1-thread-3: successes=[40], failures=[0], duration=[696ms]
Check the HTTP connection from echo client to echo server
connection info from netstat:
[root#echoclient-84485fbc5c-zxlv8 /]# netstat -ano | grep 8080 | grep ESTABLISHED
tcp 0 0 10.244.2.79:58074 10.244.2.65:8080 ESTABLISHED off (0.00/0/0)
tcp 0 0 10.244.2.79:38076 10.244.2.73:8080 ESTABLISHED off (0.00/0/0)
tcp 0 0 10.244.2.79:58088 10.244.2.65:8080 ESTABLISHED off (0.00/0/0)
tcp 0 0 10.244.2.79:38080 10.244.2.73:8080 ESTABLISHED off (0.00/0/0)
tcp 0 0 10.244.2.79:58056 10.244.2.65:8080 ESTABLISHED off (0.00/0/0)
tcp 0 0 10.244.2.79:38094 10.244.2.73:8080 ESTABLISHED off (0.00/0/0)
tcp 0 0 10.244.2.79:38110 10.244.2.73:8080 ESTABLISHED off (0.00/0/0)
tcp 0 0 10.244.2.79:58076 10.244.2.65:8080 ESTABLISHED off (0.00/0/0)
connection info from envoy cluster:
[root#echoclient-84485fbc5c-zxlv8 /]# curl -s http://localhost:15000/clusters | grep echoserver
outbound|8080||echoserver.default.svc.cluster.local::default_priority::max_connections::2
outbound|8080||echoserver.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|8080||echoserver.default.svc.cluster.local::default_priority::max_requests::1024
outbound|8080||echoserver.default.svc.cluster.local::default_priority::max_retries::3
outbound|8080||echoserver.default.svc.cluster.local::high_priority::max_connections::1024
outbound|8080||echoserver.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|8080||echoserver.default.svc.cluster.local::high_priority::max_requests::1024
outbound|8080||echoserver.default.svc.cluster.local::high_priority::max_retries::3
outbound|8080||echoserver.default.svc.cluster.local::added_via_api::true
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::cx_active::4
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::cx_connect_fail::0
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::cx_total::4
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::rq_active::0
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::rq_error::0
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::rq_success::200
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::rq_timeout::0
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::rq_total::200
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::health_flags::healthy
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::weight::1
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::region::
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::zone::
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::sub_zone::
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::canary::false
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.65:8080::success_rate::-1
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::cx_active::4
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::cx_connect_fail::0
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::cx_total::4
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::rq_active::0
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::rq_error::0
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::rq_success::200
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::rq_timeout::0
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::rq_total::200
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::health_flags::healthy
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::weight::1
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::region::
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::zone::
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::sub_zone::
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::canary::false
outbound|8080||echoserver.default.svc.cluster.local::10.244.2.73:8080::success_rate::-1
We can see that there are 8 connections from echoclient to echoserver(10.244.2.65,10.244.2.73), but not the configured maxConnections 2.
Why there are 8 connections but not 2?
Is there any misunderstanding about the maxConnections of envoy?

Answer 1

As Garrett mention in comments:
max_connections or max_requests refers to the number of connections each children of the php pool can take before it respawns and can be typically found in /etc/etc/php/{version}/fpm/pool.d/www.conf max_children is based on the memory of the machine, some example tutorials that were helpful for me to understand were Max Requests / Children

Answer 2

I think, you invoked the client with 1 thread (ie, 1 HTTP connection) but it sends requests in parallel (in batches of 10 by default)
In envoy, max_connections apply to http1 connections and in your case, you have just a single http connection.
Try taking a look at max_request which apply potentially to threads that have a close relation to http2.
Hope this helps!!

Answer 3

SSH to node of client pod, and exec into istio-proxy container
docker exec --privileged --user root -it <istio-proxy-container-id> bash
Use following command to find tcp connections envoy make to upstream
ss -pe | grep 8080 | grep envoy

Answer 4

I think envoy can make upstream connections up to max_connections per one envoy worker. If you run envoy with one concurrency option, you can check only two upstream connections are created.
$ envoy -c envoy.yaml --concurrency 1