Thrift Rate Limiting with Envoy - istio

Question

I'm extending Istio to provide Thrift features. The Istio component I'm working on right now is Pilot (Envoy config service). I've extended it with basic Thrift routing so that it can provide an Envoy configuration to route a listener to the correct cluster. My development environment looks something like:
Now I'm trying to add rate limiting. I'm following the docs. I've patched my config with a RouteAction that doesn't get applied, because there is no rate limit filter. This works fine, and passes traffic as before:
...
{
"filters": [
{
"name": "envoy.filters.network.thrift_proxy",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.filter.network.thrift_proxy.v2alpha1.ThriftProxy",
"stat_prefix": "10.97.28.169_9090",
"transport": "HEADER",
"protocol": "BINARY",
"route_config": {
"name": "outbound|9090||backend.default.svc.cluster.local",
"routes": [
{
"match": {
"method_name": ""
},
"route": {
"cluster": "outbound|9090||backend.default.svc.cluster.local",
"rate_limits": [
{
"actions": [
{
"request_headers": {
"header_name": ":method-name",
"descriptor_key": "method-name"
}
}
]
}
]
}
}
]
}
}
}
]
}
],
"deprecated_v1": {
"bind_to_port": false
},
"listener_filters_timeout": "0.100s",
"traffic_direction": "OUTBOUND",
"continue_on_listener_filters_timeout": true
},
"last_updated": "2019-10-30T16:26:39.203Z"
},
...
I've built a function to create an envoy.filters.thrift.rate_limit filter that makes Envoy to call the rate limit service I've set up (I've tried both GoogleGrpc and EnvoyGrpc):
func buildThriftRatelimit(ratelimitServiceUri, domain string) *thrift_ratelimit.RateLimit {
var thriftRateLimit *thrift_ratelimit.RateLimit
timeout := 2000 * time.Millisecond
thriftRateLimit = &thrift_ratelimit.RateLimit{
Domain: domain,
Timeout: &timeout,
FailureModeDeny: false,
RateLimitService: &ratelimit.RateLimitServiceConfig{
GrpcService: &core.GrpcService{
TargetSpecifier: &core.GrpcService_GoogleGrpc_{
GoogleGrpc: &core.GrpcService_GoogleGrpc{
StatPrefix: ratelimitServiceUri,
TargetUri: ratelimitServiceUri,
},
},
},
},
}
thriftRateLimit.Validate()
if err := thriftRateLimit.Validate(); err != nil {
panic(err)
}
return thriftRateLimit
}
Which produces:
...
{
"filters": [
{
"name": "envoy.filters.network.thrift_proxy",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.filter.network.thrift_proxy.v2alpha1.ThriftProxy",
"stat_prefix": "10.97.28.169_9090",
"transport": "HEADER",
"protocol": "BINARY",
"route_config": {
"name": "outbound|9090||backend.default.svc.cluster.local",
"routes": [
{
"match": {
"method_name": ""
},
"route": {
"cluster": "outbound|9090||backend.default.svc.cluster.local",
"rate_limits": [
{
"actions": [
{
"request_headers": {
"header_name": ":method-name",
"descriptor_key": "method-name"
}
}
]
}
]
}
}
]
},
"thrift_filters": [
{
"name": "envoy.filters.thrift.rate_limit",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.filter.thrift.rate_limit.v2alpha1.RateLimit",
"domain": "backend.default.svc.cluster.local",
"timeout": "2s",
"rate_limit_service": {
"grpc_service": {
"google_grpc": {
"target_uri": "istio-lyft-ratelimit.istio-system.svc.cluster.local:80",
"stat_prefix": "istio-lyft-ratelimit.istio-system.svc.cluster.local:80"
}
}
}
}
}
]
}
}
]
}
],
"deprecated_v1": {
"bind_to_port": false
},
"listener_filters_timeout": "0.100s",
"traffic_direction": "OUTBOUND",
"continue_on_listener_filters_timeout": true
},
"last_updated": "2019-10-30T16:26:39.203Z"
},
...
When the rate limit filter is applied, connections to the backend die and no error is returned to the client or displayed in Envoy's logs.

Answer 1

If Thrift filters are provided, you need to add the Router filter as the last filter in the chain like so:
...
{
"filters": [
{
"name": "envoy.filters.network.thrift_proxy",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.filter.network.thrift_proxy.v2alpha1.ThriftProxy",
"stat_prefix": "10.97.28.169_9090",
"transport": "HEADER",
"protocol": "BINARY",
"route_config": {
"name": "outbound|9090||backend.default.svc.cluster.local",
"routes": [
{
"match": {
"method_name": ""
},
"route": {
"cluster": "outbound|9090||backend.default.svc.cluster.local",
"rate_limits": [
{
"actions": [
{
"request_headers": {
"header_name": ":method-name",
"descriptor_key": "method-name"
}
}
]
}
]
}
}
]
},
"thrift_filters": [
{
"name": "envoy.filters.thrift.rate_limit",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.filter.thrift.rate_limit.v2alpha1.RateLimit",
"domain": "backend.default.svc.cluster.local",
"timeout": "2s",
"rate_limit_service": {
"grpc_service": {
"google_grpc": {
"target_uri": "istio-lyft-ratelimit.istio-system.svc.cluster.local:80",
"stat_prefix": "istio-lyft-ratelimit.istio-system.svc.cluster.local:80"
}
}
}
}
},
{
"name": "envoy.filters.thrift.router"
}
]
}
}
]
}
],
"deprecated_v1": {
"bind_to_port": false
},
"listener_filters_timeout": "0.100s",
"traffic_direction": "OUTBOUND",
"continue_on_listener_filters_timeout": true
},
"last_updated": "2019-10-30T16:26:39.203Z"
},
...