Envoy based header to metadata filtering regex not working - istio

My use case is to remove query parameters from the path so the envoy ISTIO filter can filter on the basis of just APIs.
I am using the below configuration it is a filtering route but also takes query parameters in the path not truncating it.
The ratelimiter service on its part does not detect any special configuration for the descriptor ("PATH", "/foo?param=value") and therfore use the default of key "PATH".
any idea why truncating regex is not working? Thanks
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: {{ template "name" . }}-httpfilter
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
subFilter:
name: "envoy.filters.http.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.header_to_metadata
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.header_to_metadata.v3.Config
request_rules:
- header: ':path'
on_header_present:
# use an arbitary name for the namespace
# will be used later to extract descriptor value
metadata_namespace: qry-filter
# use an arbitary key for the metadata
# will be used later to extract descriptor value
key: uri
regex_value_rewrite:
pattern:
# regex matcher
# truncates parameters from path
regex: '^(\/[\/\d\w-]+)\??.*$'
substitution: '\\1'
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: 'envoy.filters.network.http_connection_manager'
subFilter:
name: 'envoy.filters.http.router'
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.ratelimit
typed_config:
'#type': type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
# ensure the domain matches with the domain used in the ratelimit service config
domain: {{ template "fullname" . }}-ratelimit
failure_mode_deny: true
rate_limit_service:
grpc_service:
envoy_grpc:
# must match load_assignment.cluster_name from the patch to the CLUSTER above
cluster_name: rate_limit_cluster
timeout: 10s
transport_api_version: V3
- applyTo: CLUSTER
match:
cluster:
# kubernetes dns of your ratelimit service
service: ratelimit.{{ .Values.openapi.destinationSuffix }}
patch:
operation: ADD
value:
name: rate_limit_cluster
type: STRICT_DNS
connect_timeout: 10s
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
# arbitrary name
cluster_name: rate_limit_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
# kubernetes dns of your ratelimit service
address: ratelimit.{{ .Values.openapi.destinationSuffix }}
port_value: 8081
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: {{ template "name" . }}-virtualhost
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: VIRTUAL_HOST
match:
context: GATEWAY
routeConfiguration:
vhost:
name: ""
route:
action: ANY
patch:
operation: MERGE
value:
rate_limits:
- actions: # any actions in here
- dynamic_metadata:
descriptor_key: PATH
metadata_key:
key: qry-filter
path:
- key: uri
apiVersion: v1
kind: ConfigMap
metadata:
name: ratelimit-config
data:
config.yaml: |
domain: {{ template "fullname" . }}-ratelimit
descriptors:
- key: PATH
rate_limit:
unit: minute
requests_per_unit: 10

Related

istio oauth filter error with secrets creation - updating listener(s) 0.0.0.0_8443: paths must refer to an existing path in the system does not exist

I'm running istio on kubernetes (container istio/proxyv2:1.13.2) and currently use oauth2-proxy pod to authenticate with keycloak. I have a requirement to replace oauth2-proxy with an istio oauth filter, and I'm attempting to deploy the oauth filter to the istio ingressgateway by following this blog. When deploying the YAML below I get see the following error in istiod logs:
2022-05-12T16:59:58.080449Z warn ads ADS:LDS: ACK ERROR istio-ingressgateway-7fd568fc99-fvvcc.istio-system-150 Internal:Error adding/updating listener(s) 0.0.0.0_8443: paths must refer to an existing path in the system: '/etc/istio/config/token-secret.yaml' does not exist
It looks to me that there is an issue loading the secret YAML files into the ingressgateway pod using SDS - but I could be wrong on this as I dont fully understand how this secret loading should be working in this example. I cant find documentation on this in the latest istio versions so as such I'm struggling. Older docs talk about an sds container running in the istiogateway pod but this doesn't seem to be relevant in the more recent istio versions.
Can anyone help, or explain how the secrets get loaded into ingressgateway in the example that I'm following, and what the issue might be/how to diagnose ? Any help gratefully received.
The code is as follows:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: oauth2-ingress
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: CLUSTER
match:
cluster:
service: oauth
patch:
operation: ADD
value:
name: oauth
dns_lookup_family: V4_ONLY
type: LOGICAL_DNS
connect_timeout: 10s
lb_policy: ROUND_ROBIN
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"#type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: keycloak.mydomain.com
load_assignment:
cluster_name: oauth
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: keycloak.mydomain.com
port_value: 443
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
subFilter:
name: "envoy.filters.http.jwt_authn"
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.oauth2
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.oauth2.v3.OAuth2
config:
token_endpoint:
cluster: oauth
uri: https://keycloak.mydomain.com/auth/realms/myrealm/protocol/openid-connect/token
timeout: 3s
authorization_endpoint: https://keycloak.mydomain.com/auth/realms/myrealm/protocol/openid-connect/auth
redirect_uri: "https://%REQ(:authority)%/callback"
redirect_path_matcher:
path:
exact: /callback
signout_path:
path:
exact: /signout
credentials:
client_id: myclient
token_secret:
name: token
sds_config:
path: "/etc/istio/config/token-secret.yaml"
hmac_secret:
name: hmac
sds_config:
path: "/etc/istio/config/hmac-secret.yaml"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: istio-oauth2
namespace: istio-system
data:
token-secret.yaml: |-
resources:
- "#type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"
name: token
generic_secret:
secret:
inline_string: "myclientsecrettext"
hmac-secret.yaml: |-
resources:
- "#type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"
name: hmac
generic_secret:
secret:
# generated using `head -c 32 /dev/urandom | base64`
inline_bytes: "XYJ7ibKwXwmRrO/yL/37ZV+T3Q/WB+xfhmVlio+wmc0="
---
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: jwt-authentication
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
jwtRules:
- issuer: "https://keycloak.mydomain.com/auth/realms/myrealm"
jwksUri: "https://keycloak.mydomain.com/auth/realms/myrealm/protocol/openid-connect/certs"
forwardOriginalToken: true
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: known-user
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
rules:
- when:
- key: request.headers[Authorization]
notValues:
- 'Bearer*'
- when:
- key: request.auth.audiences
values:
- 'oauth'
- key: request.auth.presenter
values:

Match HTTP_FILTER to a specific service in Istio

I have a EnvoyFilter like below, I dont have a workload selector and need this http filter to be applied to all the sidecars. But I want the http filter to apply only when specific service/service entry is called. Is there a way to do this ?
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: headers-envoy-filter
namespace: istio-system
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_OUTBOUND
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
subFilter:
name: envoy.filters.http.router
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
'#type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inline_code: |
function envoy_on_request(request_handle)
request_handle:headers():add("custom-header", "hello")
end
function envoy_on_response(response_handle)
response_handle:headers():add("custom-header", "hello")
end

Istio rate limit support regex for the URL

Is there any way to mention the prefix of the url for rate limit in istio?
In the below config we are using /actuator/info in which is there any way to mention prefix of the URL?
Say
- key: PATH_PREFIX
value: "/actuator/"
To match all the endpoint under /actuator/* ?
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ratelimit-config
namespace: istio-system
data:
config.yaml: |
domain: test-istio-rate-limit.com
descriptors:
- key: PATH
value: "/actuator/info"
rate_limit:
unit: minute
requests_per_unit: 1
- key: PATH
rate_limit:
unit: minute
requests_per_unit: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: filter-ratelimit
namespace: istio-system
spec:
workloadSelector:
# select by label in the same namespace
labels:
istio: ingressgateway
configPatches:
# The Envoy config you want to modify
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
subFilter:
name: "envoy.filters.http.router"
patch:
operation: INSERT_BEFORE
# Adds the Envoy Rate Limit Filter in HTTP filter chain.
value:
name: envoy.filters.http.ratelimit
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
# domain can be anything! Match it to the ratelimter service config
domain: test-istio-rate-limit.com
failure_mode_deny: true
timeout: 10s
rate_limit_service:
grpc_service:
envoy_grpc:
cluster_name: rate_limit_cluster
transport_api_version: V3
- applyTo: CLUSTER
match:
cluster:
service: ratelimit.istio-system.svc.cluster.local
patch:
operation: ADD
# Adds the rate limit service cluster for rate limit service defined in step 1.
value:
name: rate_limit_cluster
type: STRICT_DNS
connect_timeout: 10s
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
cluster_name: rate_limit_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: ratelimit.istio-system.svc.cluster.local
port_value: 8081
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: filter-ratelimit-svc
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: VIRTUAL_HOST
match:
context: GATEWAY
routeConfiguration:
vhost:
name: ""
route:
action: ANY
patch:
operation: MERGE
# Applies the rate limit rules.
value:
rate_limits:
- actions: # any actions in here
- request_headers:
header_name: ":path"
descriptor_key: "PATH"
Use https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/header_to_metadata/v3/header_to_metadata.proto.html
Example : https://dev.to/tresmonauten/setup-an-ingress-rate-limiter-with-envoy-and-istio-1i9g , scroll down to 'Beware of pitfalls'

how to use hcm filter in envoy filter?

guys! i want to distribute flow in egress by using hcm, such as
www.xxx.com:8000(sidecar) -> clusterA(egress)
www.yyy.com:8000(sidecar) -> clusterB(egress)
rest request (sidecar) -> original process(egress)
the match request can be routed correctly,but how about the unmatched request?
the unmatched request get 404 response now.
i want to unmatched request to be processed in the original path.
so how can i modify yaml configure to achieve this goal ??does there have best practice?
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: test
namespace: istio-system
spec:
workloadSelector:
labels:
istio: egressgateway
configPatches:
- applyTo: NETWORK_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
patch:
operation: REPLACE
value:
name: envoy.filters.network.http_connection_manager
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
codec_type: AUTO
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service1
domains: ["www.xxx.com:8000"]
routes:
- match:
prefix: "/"
route:
host_rewrite_literal: www.xxx.com:80
prefix_rewrite: "/"
cluster: "clusterA"
- name: local_service2
domains: ["www.yyy.com:8000"]
routes:
- match:
prefix: "/"
route:
host_rewrite_literal: www.yyy.com:80
prefix_rewrite: "/"
cluster: "clusterB"
http_filters:
- name: envoy.filters.http.router
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
#... other configure here.
# cluster configure

Istio response compression filter warning

For some reason, Istio 1.8.1 stopped accepting gzip filter for Envoy.
Warning: Envoy filter: can't unmarshal Any nested proto type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor: unknown field "response_direction_config" in envoy.extensions.filters.http.compressor.v3.Compressor
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: gzip
spec:
workloadSelector:
labels:
app: istio-ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: envoy.http_connection_manager
subFilter:
name: envoy.router
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.compressor
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
remove_accept_encoding_header: true
response_direction_config:
common_config:
min_content_length: 100
content_type:
- '*/*'
compressor_library:
name: for_response
typed_config:
"#type": type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
memory_level: 9
window_bits: 15
compression_level: 9