My use case is to remove query parameters from the path so the envoy ISTIO filter can filter on the basis of just APIs.
I am using the below configuration it is a filtering route but also takes query parameters in the path not truncating it.
The ratelimiter service on its part does not detect any special configuration for the descriptor ("PATH", "/foo?param=value") and therfore use the default of key "PATH".
any idea why truncating regex is not working? Thanks
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: {{ template "name" . }}-httpfilter
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
subFilter:
name: "envoy.filters.http.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.header_to_metadata
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.header_to_metadata.v3.Config
request_rules:
- header: ':path'
on_header_present:
# use an arbitary name for the namespace
# will be used later to extract descriptor value
metadata_namespace: qry-filter
# use an arbitary key for the metadata
# will be used later to extract descriptor value
key: uri
regex_value_rewrite:
pattern:
# regex matcher
# truncates parameters from path
regex: '^(\/[\/\d\w-]+)\??.*$'
substitution: '\\1'
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: 'envoy.filters.network.http_connection_manager'
subFilter:
name: 'envoy.filters.http.router'
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.ratelimit
typed_config:
'#type': type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
# ensure the domain matches with the domain used in the ratelimit service config
domain: {{ template "fullname" . }}-ratelimit
failure_mode_deny: true
rate_limit_service:
grpc_service:
envoy_grpc:
# must match load_assignment.cluster_name from the patch to the CLUSTER above
cluster_name: rate_limit_cluster
timeout: 10s
transport_api_version: V3
- applyTo: CLUSTER
match:
cluster:
# kubernetes dns of your ratelimit service
service: ratelimit.{{ .Values.openapi.destinationSuffix }}
patch:
operation: ADD
value:
name: rate_limit_cluster
type: STRICT_DNS
connect_timeout: 10s
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
# arbitrary name
cluster_name: rate_limit_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
# kubernetes dns of your ratelimit service
address: ratelimit.{{ .Values.openapi.destinationSuffix }}
port_value: 8081
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: {{ template "name" . }}-virtualhost
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: VIRTUAL_HOST
match:
context: GATEWAY
routeConfiguration:
vhost:
name: ""
route:
action: ANY
patch:
operation: MERGE
value:
rate_limits:
- actions: # any actions in here
- dynamic_metadata:
descriptor_key: PATH
metadata_key:
key: qry-filter
path:
- key: uri
apiVersion: v1
kind: ConfigMap
metadata:
name: ratelimit-config
data:
config.yaml: |
domain: {{ template "fullname" . }}-ratelimit
descriptors:
- key: PATH
rate_limit:
unit: minute
requests_per_unit: 10
I have a EnvoyFilter like below, I dont have a workload selector and need this http filter to be applied to all the sidecars. But I want the http filter to apply only when specific service/service entry is called. Is there a way to do this ?
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: headers-envoy-filter
namespace: istio-system
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_OUTBOUND
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
subFilter:
name: envoy.filters.http.router
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
'#type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inline_code: |
function envoy_on_request(request_handle)
request_handle:headers():add("custom-header", "hello")
end
function envoy_on_response(response_handle)
response_handle:headers():add("custom-header", "hello")
end
Is there any way to mention the prefix of the url for rate limit in istio?
In the below config we are using /actuator/info in which is there any way to mention prefix of the URL?
Say
- key: PATH_PREFIX
value: "/actuator/"
To match all the endpoint under /actuator/* ?
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ratelimit-config
namespace: istio-system
data:
config.yaml: |
domain: test-istio-rate-limit.com
descriptors:
- key: PATH
value: "/actuator/info"
rate_limit:
unit: minute
requests_per_unit: 1
- key: PATH
rate_limit:
unit: minute
requests_per_unit: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: filter-ratelimit
namespace: istio-system
spec:
workloadSelector:
# select by label in the same namespace
labels:
istio: ingressgateway
configPatches:
# The Envoy config you want to modify
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
subFilter:
name: "envoy.filters.http.router"
patch:
operation: INSERT_BEFORE
# Adds the Envoy Rate Limit Filter in HTTP filter chain.
value:
name: envoy.filters.http.ratelimit
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
# domain can be anything! Match it to the ratelimter service config
domain: test-istio-rate-limit.com
failure_mode_deny: true
timeout: 10s
rate_limit_service:
grpc_service:
envoy_grpc:
cluster_name: rate_limit_cluster
transport_api_version: V3
- applyTo: CLUSTER
match:
cluster:
service: ratelimit.istio-system.svc.cluster.local
patch:
operation: ADD
# Adds the rate limit service cluster for rate limit service defined in step 1.
value:
name: rate_limit_cluster
type: STRICT_DNS
connect_timeout: 10s
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
cluster_name: rate_limit_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: ratelimit.istio-system.svc.cluster.local
port_value: 8081
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: filter-ratelimit-svc
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: VIRTUAL_HOST
match:
context: GATEWAY
routeConfiguration:
vhost:
name: ""
route:
action: ANY
patch:
operation: MERGE
# Applies the rate limit rules.
value:
rate_limits:
- actions: # any actions in here
- request_headers:
header_name: ":path"
descriptor_key: "PATH"
Use https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/header_to_metadata/v3/header_to_metadata.proto.html
Example : https://dev.to/tresmonauten/setup-an-ingress-rate-limiter-with-envoy-and-istio-1i9g , scroll down to 'Beware of pitfalls'
I'm trying to make EnvoyFilters work in my installation.
For test purposes I'm trying to set lua filter that logs dumb message and adds header to the resonse.
Here's my configuration:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: dumb-filter
namespace: istio-system
spec:
# workloadSelector:
# labels:
# istio: ingressgateway
configPatches:
# - applyTo: VIRTUAL_HOST
- applyTo: HTTP_ROUTE
match:
context: GATEWAY
# context: ANY
routeConfiguration:
vhost:
# name: "<domain>:443"
route:
#TODO: Understand name compose logic
name: https.443.https.geth-dedicated.default
patch:
operation: MERGE
value:
name: envoy.filters.http.lua
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inlineCode: |
function envoy_on_response(response_handle)
response_handle:headers():add("dm3ch-test", "dm3ch")
response_handle:logErr("Bye Bye.")
end
For now I see no log message or test header in response.
I already tried:
create EnvoyFilter object in both application and istio-system namespace (where istio gateway pods lives)
specifying workloadSelector (I have verified that istio gateway pod have istio: ingressgateway label)
changing context from "GATEWAY" to "ANY"
changing applyTo to VIRTUAL_HOST and HTTP_ROUTE modes
verified that route name is actually https.443.https.geth-dedicated.default using istioctl proxy-config route <gateway_pod> command.
adding vhost.name setting and commenting vhost.route.name
Istio version info:
❯ istioctl version
client version: 1.11.4
control plane version: 1.12.0-alpha.1
data plane version: 1.12.0-alpha.1 (1 proxies)
route configuration json:
❯ istioctl proxy-config route istio-ingress-675cb54bc9-5r8cs.istio-system --name https.443.https.geth-dedicated.default -o json
[
{
"name": "https.443.https.geth-dedicated.default",
"virtualHosts": [
{
"name": "<domain>:443",
"domains": [
"<domain>",
"<domain>:*"
],
"routes": [
{
"match": {
"prefix": "/",
"caseSensitive": true
},
"route": {
"cluster": "outbound|8545||geth-dedicated.default.svc.cluster.local",
"timeout": "0s",
"retryPolicy": {
"retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"numRetries": 2,
"retryHostPredicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
"hostSelectionRetryMaxAttempts": "5",
"retriableStatusCodes": [
503
]
},
"hashPolicy": [
{
"connectionProperties": {
"sourceIp": true
}
}
],
"maxGrpcTimeout": "0s"
},
"metadata": {
"filterMetadata": {
"istio": {
"config": "/apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/geth-dedicated"
}
}
},
"decorator": {
"operation": "geth-dedicated.default.svc.cluster.local:8545/*"
}
}
],
"includeRequestAttemptCount": true
}
],
"validateClusters": false
I would be glad if anyone could consult me what am I doing wrong or how can I better debug why filter is not applied.
P.S. My goal is to invoke custom logic during request/response handling on ingressgateway istio deployment for specific virtualservice only
Chris answer was very useful, but unfortunately it wasn't not full. :(
Here what I've found:
It's not possible to use type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua filter on HTTP_ROUTE (but it's possible to use LuaPerRoute)
type.googleapis.com/envoy.extensions.filters.http.lua.v3.LuaPerRoute itself doesn't allow to define new lua filter, it allows only disable existing Lua filter or override it's source code envoy docs
So to make lua custom logic that is applied to only one http route you need to define "global" Lua filter and override it's code for specific http route using LuaPerRoute filter.
Here's my manifests that allowed me to make it work:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: geth-dedicated
namespace: default
spec:
gateways:
- geth-dedicated # I'm ommiting gateway creation in this snippet
hosts:
- <domain>
http:
- match:
- uri:
prefix: /
name: geth-public
route:
- destination:
host: geth-dedicated
port:
number: 8545
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: dumb-filter
namespace: istio-system # Namespace where istio gateway pods are actually running
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
# Patch that creates "global" lua filter that does nothing useful
- applyTo: HTTP_FILTER
match:
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
subFilter:
name: envoy.filters.http.router
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
'#type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inlineCode: |
function envoy_on_request(request_handle)
-- Empty lua function
end
# Filter for http route that overrides "global" filter lua source code
- applyTo: HTTP_ROUTE
match:
context: GATEWAY
routeConfiguration:
vhost:
route:
name: geth-public # Corresponds to http[0].name in VirtualService
patch:
operation: MERGE
value:
name: envoy.lua
typed_per_filter_config:
envoy.filters.http.lua:
'#type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.LuaPerRoute
source_code:
inline_string: |
function envoy_on_response(response_handle)
response_handle:logErr("Goodbye my brain.")
response_handle:headers():add("dm3ch-test", "dm3ch wins")
end
The problem is your todo #TODO: Understand name compose logic. You need to set this name value to the name of the route of the VirtualService. Also you need to use a typed_per_filter_config with a type LuaPerRoute.
If your VirtualService looks something like that:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: reviews-route
spec:
hosts:
- reviews.prod.svc.cluster.local
http:
- name: "reviews-v2-routes"
route:
- destination:
host: reviews.prod.svc.cluster.local
subset: v2
- name: "reviews-v1-route"
route:
- destination:
host: reviews.prod.svc.cluster.local
subset: v1
than your EnvoyFilter needs to be setup like this:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: dumb-filter
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_ROUTE
match:
routeConfiguration:
vhost:
route:
# name from virtual service route that the filter should apply to
name: reviews-v1-route
patch:
operation: MERGE
value:
# 'custom' as prefix, can be anything
name: custom.dumb-filter
# set lua per route filter
typed_per_filter_config:
envoy.filters.http.lua:
"#type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.LuaPerRoute
source_code:
inline_string: |
function envoy_on_response(response_handle)
response_handle:headers():add("dm3ch-test", "dm3ch")
response_handle:logErr("Bye Bye.")
end
Note:
This requires a lua filter that is already applied because the LuaPerRoute will only overwrite an existing one.
I am using envoy external auth filter in istio. It works version 1.6 and 1.7. But in version 1.8 it is not working anymore. I want to use version 1.8 but i could not manage to work with this version.
What should be problem ?
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: authn-filter
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
subFilter:
name: "envoy.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.ext_authz
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
clear_route_cache: true
http_service:
server_uri:
uri: http://auth-http-service.default.svc.cluster.local
cluster: outbound|3000||auth-http-service.default.svc.cluster.local
timeout: 1.5s
authorizationRequest:
allowedHeaders:
patterns:
- exact: "cookie"
authorizationResponse:
allowedClientHeaders:
patterns:
- exact: "authorization"
allowedUpstreamHeaders:
patterns:
- exact: "set-cookie"
- exact: "authorization"