I have a setup using Kubernetes and Istio where we run a set of services. Each of our services have an istio-sidecar and a REST-api. What we would like is that whenever a service within our setup calls another that the called service knows what service is the caller (Preferably through a header).
Looking at the example image from bookinfo:
bookinfo-image (Link due to <10 reputation)
This would mean that in the source code for the ratings service I would like to be able to, for example, read a header telling me the request came from e.g. Reviews-v2.
My intuition tells me that I should be able to handle this in the istio sidecars, but I fail to realise exactly how.
Until now I have looked at especially envoy filters in the hope that they could help me. I see that for the envoy filters I would be able to set a header, but what I don't see is how I would get the information about what service made the call in order to set it in the header.
Envoy automatically sets the X-Forwarded-Client-Cert header, which contains the SPIFFE ID of the caller. SPIFFE ID in Istio is a URI in the form spiffe://cluster.local/ns/<namespace>/sa/<service account>. Practically, it designates the Kubernetes Service Account of the caller. You may want to test it by using the Istio httpbin sample and sending a request to httpbin:8000/headers
I ended up finding another solution by using a "rule". If we made sure that policy enforcing is enabled and then added the rule:
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
name: header-rule
namespace: istio-system
spec:
actions: []
requestHeaderOperations:
- name: serviceid
values:
- source.labels["app"]
operation: REPLACE
We achieved what we were attempting to do.
Related
We have deployed Istio 1.11.0 using helm-chart in our dev and production environment.
We are using below configuration in istio configmap, which we have updated via istio-control helm-chart.
meshConfig:
extensionProviders:
- name: "ext-authz-grpc"
envoyExtAuthzGrpc:
service: "ext-auth-service.default.svc.cluster.local"
port: "50051"
includeHeadersInCheck: [ "authorization", "ws-protocol" ]
headersToUpstreamOnAllow: [ "authorization", "x-role", "x-id" ]
accessLogFile: /dev/stdout
enablePrometheusMerge: true
Basically we are using grpc service for external authorization server.
Above configuration is working fine.
One of our client has deployed Istio 1.9.8 using operator. (They have their own deployment model for Istio. Not allowing us to deploy istio using helm-chart)
When we try to apply above changes using operator it gives us below error :
2022-04-05T10:23:09.657830Z info installer Loading values from compiled in VFS at path profiles/minimal.yaml
2022-04-05T10:23:09.657837Z info installer Loading values from compiled in VFS at path profiles/default.yaml
2022-04-05T10:23:09.679340Z error installer failed to merge base profile with user IstioOperator CR profile-poc-customized, failed to unmarshall mesh config: unknown field "includeHeadersInCheck" in v1alpha1.MeshConfig_ExtensionProvider_EnvoyExternalAuthorizationGrpcProvider moreInfo=The values in the selected spec.profile could not be merged with the user IstioOperator resource. impact=The operator controller cannot create and act upon the user defined IstioOperator resource. The Istio control plane will not be installed or updated. action=Check that the IstioOperator resource has the correct syntax. If you are sure your configuration is correct, see https://istio.io/latest/about/bugs for possible solutions. likelyCause=The likely cause is an incorrect or badly formatted configuration.Another possible cause could be an issue with the Istio code.
If we directly edit the configmap and make changes then it is able to apply those changes.
But its giving error when we are updating it from operator.
Can anybody help me to understand why its not working with operator?
includeHeadersInCheck is only available for http and not grpc:
https://istio.io/v1.10/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-ExtensionProvider-EnvoyExternalAuthorizationGrpcProvider
I am setting up a GCP url map to route requests to backend services based on cookie values. Since cookies would have multiple key values, I am trying to use a regex matcher.
I need to route requests to backends based on region value from cookie.
A typical cookie would look like this: foo=bar;region=eu;variant=beta;
defaultService: https://www.googleapis.com/compute/v1/projects/<project_id>/global/backendServices/multi-region-1
kind: compute#urlMap
name: regex-url-map
hostRules:
- hosts:
- '*'
pathMatcher: path-matcher-1
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/<project_id>/global/backendServices/multi-region-1
name: path-matcher-1
routeRules:
- matchRules:
- prefixMatch: /
headerMatches:
- headerName: Cookie
regexMatch: (region=us)
priority: 0
service: https://www.googleapis.com/compute/v1/projects/<project_id>/global/backendServices/multi-region-1
- matchRules:
- prefixMatch: /
headerMatches:
- headerName: Cookie
regexMatch: (region=eu)
priority: 1
service: https://www.googleapis.com/compute/v1/projects/<project_id>/global/backendServices/multi-region-2
However, this url-map fails validation with this error:
$ gcloud compute url-maps validate --source regex-url-map.yaml
result:
loadErrors:
- HttpHeaderMatch has no predicates specified
loadSucceeded: false
testPassed: false
Please note that an exact match with cookie passes validation and matches correctly if cookie value is just something like this: region=us. The headerMatches section for exact match would look like this:
headerMatches:
- headerName: Cookie
exactMatch: region=us
Any pointers on what am I doing wrong here?
Thanks!
Your way of reasoning is correct but the feature you're trying to use is unsupported in external load balancing in GCP; it works only with internal load balancing.
Look at the last phrase from the documentation:
Note that regexMatch only applies to Loadbalancers that have their loadBalancingScheme set to INTERNAL_SELF_MANAGED.
I know it isn't the answer you're looking for but you can always file a new feature request on Google's IssueTracker and explain in detail what you want, how it could work etc.
You can always try to pass the region value in the http request - instead of requesting https://myhost.com all the time - also if you could add a suffix, for example: https://myhost.com/region1 it would allow the GCP load balancer rules to process it and direct the traffic to the backend you wish.
Have a look at this example what you can and can't do with forwarding rules in GCP. Another example here. And another one (mine) explaining how to use pathMatcher to direct traffic to different backend services.
Istio supports fault injecting.
The example is using destination. Is there any way for me to use source in order to inject fault to downstream services?
I would suggest using sourceLabels with internal mesh gateway.
Example:
gateways:
- mesh
http:
- match:
- sourceLabels:
app: source-app-v1
read HTTPMatchRequest for more.
Unfortunately, there is no straightforward way to achieve what you looking for.
route.destination.host is a required field for HTTPFaultInjection, and it has to be unique [source], and unambiguously refer to a service in the service registry [source], so it cannot be wildcarded (with * for example).
I am using istio with version 1.3.5. Is there any configuration to be set to allow istio-proxy to log traceId? I am using jaeger tracing (wit zipkin protocol) being enabled. There is one thing I want to accomplish by having traceId logging:
- log correlation in multiple services upstream. Basically I can filter all logs by certain traceId.
According to envoy proxy documentation for envoy v1.12.0 used by istio 1.3:
Trace context propagation
Envoy provides the capability for reporting tracing information regarding communications between services in the mesh. However, to be able to correlate the pieces of tracing information generated by the various proxies within a call flow, the services must propagate certain trace context between the inbound and outbound requests.
Whichever tracing provider is being used, the service should propagate the x-request-id to enable logging across the invoked services to be correlated.
The tracing providers also require additional context, to enable the parent/child relationships between the spans (logical units of work) to be understood. This can be achieved by using the LightStep (via OpenTracing API) or Zipkin tracer directly within the service itself, to extract the trace context from the inbound request and inject it into any subsequent outbound requests. This approach would also enable the service to create additional spans, describing work being done internally within the service, that may be useful when examining the end-to-end trace.
Alternatively the trace context can be manually propagated by the service:
When using the LightStep tracer, Envoy relies on the service to propagate the
x-ot-span-context
HTTP header while sending HTTP requests to other services.
When using the Zipkin tracer, Envoy relies on the service to propagate the B3 HTTP headers (
x-b3-traceid,
x-b3-spanid,
x-b3-parentspanid,
x-b3-sampled,
and
x-b3-flags).
The
x-b3-sampled
header can also be supplied by an external client to either enable or
disable tracing for a particular request. In addition, the single
b3
header propagation format is supported, which is a more compressed
format.
When using the Datadog tracer, Envoy relies on the service to propagate the Datadog-specific HTTP headers (
x-datadog-trace-id,
x-datadog-parent-id,
x-datadog-sampling-priority).
TLDR: traceId headers need to be manually added to B3 HTTP headers.
Additional information: https://github.com/openzipkin/b3-propagation
I saw Istio site mention Rate Limiting support but I can only find global rate-limit example.
Is it possible to do so at user-level? For example, if my user logged in but sends more than 50 requests within a second then I'd like to block said user, etc. In a similar situation, if user doesn't logged in then that device cannot send more than 30 requests per seconds.
Yes it is possible to conditionally apply rate limits based on arbitrary
attributes using a match condition in the quota rule.
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
name: quota
namespace: istio-system
spec:
match: source.namespace != destination.namespace
actions:
- handler: handler.memquota
instances:
- requestcount.quota
The quota will only apply when the source namespace is not equal to the destination namespace. In your case you probablty want to set a match like this:
match:
request:
headers:
cookie:
regex: "^(.*?;)?(user=jason)(;.*)?$"
I made a PR to improve the rate-limiting docs you can find it here: https://github.com/istio/istio.github.io/pull/1109