We have deployed Istio 1.11.0 using helm-chart in our dev and production environment.
We are using below configuration in istio configmap, which we have updated via istio-control helm-chart.
meshConfig:
extensionProviders:
- name: "ext-authz-grpc"
envoyExtAuthzGrpc:
service: "ext-auth-service.default.svc.cluster.local"
port: "50051"
includeHeadersInCheck: [ "authorization", "ws-protocol" ]
headersToUpstreamOnAllow: [ "authorization", "x-role", "x-id" ]
accessLogFile: /dev/stdout
enablePrometheusMerge: true
Basically we are using grpc service for external authorization server.
Above configuration is working fine.
One of our client has deployed Istio 1.9.8 using operator. (They have their own deployment model for Istio. Not allowing us to deploy istio using helm-chart)
When we try to apply above changes using operator it gives us below error :
2022-04-05T10:23:09.657830Z info installer Loading values from compiled in VFS at path profiles/minimal.yaml
2022-04-05T10:23:09.657837Z info installer Loading values from compiled in VFS at path profiles/default.yaml
2022-04-05T10:23:09.679340Z error installer failed to merge base profile with user IstioOperator CR profile-poc-customized, failed to unmarshall mesh config: unknown field "includeHeadersInCheck" in v1alpha1.MeshConfig_ExtensionProvider_EnvoyExternalAuthorizationGrpcProvider moreInfo=The values in the selected spec.profile could not be merged with the user IstioOperator resource. impact=The operator controller cannot create and act upon the user defined IstioOperator resource. The Istio control plane will not be installed or updated. action=Check that the IstioOperator resource has the correct syntax. If you are sure your configuration is correct, see https://istio.io/latest/about/bugs for possible solutions. likelyCause=The likely cause is an incorrect or badly formatted configuration.Another possible cause could be an issue with the Istio code.
If we directly edit the configmap and make changes then it is able to apply those changes.
But its giving error when we are updating it from operator.
Can anybody help me to understand why its not working with operator?
includeHeadersInCheck is only available for http and not grpc:
https://istio.io/v1.10/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-ExtensionProvider-EnvoyExternalAuthorizationGrpcProvider
Related
I am getting class cast exception when trying to setup micro gateway in APIM 2.6.0. please advise.
please advise.
command executed : ./bin/micro-gw setup hello-world -a HelloWorld -v v1
[2021-01-06 15:19:21,126] DEBUG {org.wso2.apimgt.gateway.cli.rest.RESTAPIServiceImpl} - Retrieving API with name HelloWorld, version v1 was successful. [2021-01-06 15:19:21,357] ERROR {org.wso2.apimgt.gateway.cli.cmd.Main} - Internal error occurred while executing command.java.lang.ClassCastException: org.wso2.apimgt.gateway.cli.model.rest.policy.BandwidthLimitDTO cannot be cast to org.wso2.apimgt.gateway.cli.model.rest.policy.RequestCountLimitDTO
at org.wso2.apimgt.gateway.cli.model.template.policy.ThrottlePolicy.buildContext(ThrottlePolicy.java:138)
at org.wso2.apimgt.gateway.cli.codegen.ThrottlePolicyGenerator.generateSubscriptionPolicies(ThrottlePolicyGenerator.java:97)
at org.wso2.apimgt.gateway.cli.codegen.ThrottlePolicyGenerator.generate(ThrottlePolicyGenerator.java:59)
at org.wso2.apimgt.gateway.cli.cmd.SetupCmd.execute(SetupCmd.java:298)
at java.util.Optional.ifPresent(Optional.java:159)
In this example, I had selected API that had Tier value selected as bandwidth based throttling. I have removed that and re-published this API to resolve issue. It worked after that. I do not need bandwidth based throttling right now, if someone need that then this issue will be there.Thanks
Using Istio 1.2.10-gke.3 on gke
curl -v -HHost:user.domain.com --resolve user.domain.com:443:$gatewayIP https://user.domain.com/auth -v -k
return a 503 after tls verification
< date: Tue, 19 May 2020 20:50:29 GMT
< server: istio-envoy
Now I want to track the request and identify the first point of failure by tracing the logs of the components involved and resolve the issue
The logs of the istio-ingressgateway pod show nothing. After getting a shell on the pod, I do a top and see an envoy process running, however I don't see any logs for the envoy in /var/log/
What am I missing? Am I looking at the wrong place? Or do I need to read the code of the framework to be able to use it?
I need to find out which link in the request processing chain broke first and the reason so that the same can be fixed
Here are some useful links to istio documentation for debugging error 503:
Istio documentation for envoy access logs
Istio documentation for Connectivity troubleshooting.
Useful envoy debugging tool istioctl.
$ istioctl proxy-status
Also one rare case where error 503 could be present.
This error could also be present if envoy sidecar proxy has issues or did not properly inject itself to deployment pod. Or when there are mTLS miss-configurations.
Hope it helps.
I am trying to create a lambda S3 listener leveraging Lambda as a native image. The point is to get the S3 event and then do some work by pulling the file, etc. To get the file I am using het AWS 2.x S3 client as below
S3Client.builder().httpClient().build();
This code results in
2020-03-12 19:45:06,205 ERROR [io.qua.ama.lam.run.AmazonLambdaRecorder] (Lambda Thread) Failed to run lambda: software.amazon.awssdk.core.exception.SdkClientException: Unable to load an HTTP implementation from any provider in the chain. You must declare a dependency on an appropriate HTTP implementation or pass in an SdkHttpClient explicitly to the client builder.
To resolve this I added the aws apache client and updated the code to do the following:
SdkHttpClient httpClient = ApacheHttpClient.builder().
maxConnections(50).
build()
S3Client.builder().httpClient(httpClient).build();
I also had to add:
[
["org.apache.http.conn.HttpClientConnectionManager",
"org.apache.http.pool.ConnPoolControl","software.amazon.awssdk.http.apache.internal.conn.Wrapped"]
]
After this I am now getting the following stack trace:
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120)
at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104)
at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:86)
... 76 more
I am running version 1.2.0 of qurkaus on 19.3.1 of graal. I am building this via Maven and the the provided docker container for Quarkus. I thought the trust store was added by default (in the build command it looks to be accurate) but am I missing something? Is there another way to get this to run without the setting of the HttpService on the S3 client?
There is a PR, under review at the moment, that introduces AWS S3 extension both JVM & Native. AWS clients are fully Quarkified, meaning configured via application.properties and enabled for dependency injection. So stay tuned as it most probably be available in Quarkus 1.5.0
I have multiple targets for my prometheus server but only for one I am getting the error
context deadline exceeded
Even I am able to browse the metrics URL from wmi_exporter and it just results in less a second, I tried increasing the scrape interval for this specific target but no luck.
I cannot understand if I am able to browser the wmi_exporter URL from the same machine where prometheus is running, why prometheus is still showing that error.
Please help
The solution is very easy, you only need to edit the YAML file of daemon set prometheus-node-exporter to delete hostNetwork: true. If you are using Helm chart prometheus-operator to install Prometheus Operator, you can override the hostNetwork setting using a values file with -f option.
prometheus-node-exporter:
hostNetwork: false
I have a setup using Kubernetes and Istio where we run a set of services. Each of our services have an istio-sidecar and a REST-api. What we would like is that whenever a service within our setup calls another that the called service knows what service is the caller (Preferably through a header).
Looking at the example image from bookinfo:
bookinfo-image (Link due to <10 reputation)
This would mean that in the source code for the ratings service I would like to be able to, for example, read a header telling me the request came from e.g. Reviews-v2.
My intuition tells me that I should be able to handle this in the istio sidecars, but I fail to realise exactly how.
Until now I have looked at especially envoy filters in the hope that they could help me. I see that for the envoy filters I would be able to set a header, but what I don't see is how I would get the information about what service made the call in order to set it in the header.
Envoy automatically sets the X-Forwarded-Client-Cert header, which contains the SPIFFE ID of the caller. SPIFFE ID in Istio is a URI in the form spiffe://cluster.local/ns/<namespace>/sa/<service account>. Practically, it designates the Kubernetes Service Account of the caller. You may want to test it by using the Istio httpbin sample and sending a request to httpbin:8000/headers
I ended up finding another solution by using a "rule". If we made sure that policy enforcing is enabled and then added the rule:
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
name: header-rule
namespace: istio-system
spec:
actions: []
requestHeaderOperations:
- name: serviceid
values:
- source.labels["app"]
operation: REPLACE
We achieved what we were attempting to do.