In kubernetes (AWS EKS) v1.20 have a default namespace with two pods, connected with a service type loadbalancer (CLB). Requesting the uri to the CLB worked fine and routed to either of the pods as required.
Post installation of 1.13.0 of Istio with istio-injection=enabled label set on a different namespace, the communication of the non-istio pods with no sidecar injection doesnt seem to work.
What I mean by doesnt work: (below 3 scenarios always worked without istio)
curl requests sent to https://default-nspods/apicall always worked with the non-istio pods.
i.e., the CLB always forwarded requests to to the 2 pods as required.
curl request after logging into the pod1 to pod2s IP worked and vice versa.
curl request to pod2 uri from the Node1 of pod1 worked and vice versa.
Post
Post installation, 2 and 3 doesnt work. The CLB also has trouble reading the nodeport of the pods at times.
Ive checked istioctl proxy-config endpoints and checked the deployments where the sidecar injection is enabled, the output doesnt show any other non mesh service/pod details.
Istio Version: 1.13.0
Ingress Gateway: Enabled (Loadbalancer mode)
Egress Gateway: Disabled
No addons like Kiali, Prometheus
Istio Operator based installation with modified yaml values.
Single cluster installation i.e., ISTIO_MESH_ROUTER_MODE='standard'
Istio pods, envoy sidecars, proxy-config dont show any errors.
Am kind of stuck, please let me know if I need to check kube-proxy, ip-tables or some where else.
Ive uninstalled istio using the "istioctl x uninstall --purge" option and re-installed , but the non-mesh pods seem to be not working now with Istio installed or not.
Istio pods and Istio injection namespace pods dont have issues.
Related
I built a new cluster with Terraform for a AWS EKS, single node group with a single node.
This cluster is using 1.22 and cant seem to get anything to work correctly.
So Istio will install fine, i have installed versions 1.12.1, 1.13.2, 1.13.3 & 1.13.4 and all seem to have the same issue with auto injecting the sidecar.
Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded
But there are also other issues with the cluster, even without using Istio. My application is pulled and the pod will build fine but can not connect to the database. This is an external DB to the cluster - no other build (running on Azure) have any issues connecting to the DB
I am not sure if this is the issue with the application not connecting to the ext. DB but the sidecar issue could have something to do with BoundServiceAccountTokenVolume?
There is a warming about it being enabled on all clusters from 1.21 - a little odd as i have another applications with istio, running on another cluster with 1.21 on AWS EKS!
I also have this application running with istio without any issues in Azure on 1.22
I seem to have fix it :)
It seems to be a port issue with the security groups. I was letting terraform build its own group.
When I opened all the ports up in the 'inbound' section it seemed to work.
I then closed them all again and only opened 80 and 443 - which again stopped Istio from auto-injecting its sidecar
My app was requesting to talk to Istio on port 15017, so i opened just that port, along sided ports 80 and 443.
Once that port was opened, my app started to work and got the sidecar from Istio without any issue.
So it seems like the security group stops pod-to-pod communication... unless i have completely messed up my terraform build in some way
I have some basic questions about Istio. I installed Istio for my Tyk API gateway. Then I found that simply installing Istio will cause all traffic between the Tyk pods to be blocked. Is this the default behaviour for Istio? The Tyk gateway cannot communicate with the Tyk dashboard.
When I rebuild my deployment without Istio, everything works fine.
I have also read that Istio can be configured with virtual services to perform traffic routing. Is this what I need to do for every default installing of Istio? Meaning, if I don't create any virtual services, then Istio will block all traffic by default?
Secondly, I understand a virtual service is created as a YAML file applied as a CRD. The host name defined in the virtual service rules - in a default Kubernetes cluster implementation on Google Cloud, how do I find out the host name of my application?
Lastly, if I install Tyk first, then later install Istio, and I have created the necessary label in Tyk's nanmespace for the proxy to be injected, can I just perform a rolling upgrade of my Tyk pods to have Istio start the injection?
For example, I have these labels in my Tyk dashboard service. Do I use the value called "app" in my virtual service YAML?
labels:
app: dashboard-svc-tyk-pro
app.kubernetes.io/managed-by: Helm
chart: tyk-pro-0.8.1
heritage: Helm
release: tyk-pro
Sorry for all the basic questions!
For question on Tyk gateway cannot communicate with the Tyk dashboard.
(I think the problem is that your pod tries to connect to the database before the Istio sidecar is ready. And thus the connection can't be established.
Istio runs an init container that configures the pods route table so all traffic is routed through the sidecar. So if the sidecar isn't running and the other pod tries to connect to the db, no connection can be established. Ex case: Application running in Kubernetes cron job does not connect to database in same Kubernetes cluster)
For question on Virtual Services
2.Each virtual service consists of a set of routing rules that are evaluated in order, letting Istio match each given request to the virtual service to a specific real destination within the mesh.
By default, Istio configures the Envoy proxies to passthrough requests to unknown services. However, you can’t use Istio features to control the traffic to destinations that aren’t registered in the mesh.
For question on hostname refer to this documentation.
The hosts field lists the virtual service’s hosts - in other words, the user-addressable destination or destinations that these routing rules apply to. This is the address or addresses the client uses when sending requests to the service.
Adding Istio on GKE to an existing cluster please refer to this documentation.
If you want to update a cluster with the add-on, you may need to first resize your cluster to ensure that you have enough resources for Istio. As when creating a new cluster, we suggest at least a 4 node cluster with the 2 vCPU machine type.If you have an existing application on the cluster, you can find out how to migrate it so it's managed by Istio as mentioned in the Istio documentation.
You can uninstall the add-on following document which includes to shift traffic away from the Istio ingress gateway.Please take a look at this doc for more details on installing and uninstalling Istio on GKE.
Also adding this document for installing Istio on GKE which also includes installing it to an existing cluster to quickly evaluate Istio.
I'm quite new to Istio.
I just installed Istio on a k8 cluster in GCP. I have 2 services in my private cluster. one of them needs to talk to a Redis memorystore (over internal private IP - 10.x.x.x).
I'm seeing errors trying to connect to redis. What am I missing in my Istio configuration?
Update: I have found that the redis error is misleading. The real issue it seems is something else - see one my comments below. I don't understand what that error means.
Some additional background: this is for a Tyk installation. The issue it seems is communication between the Tyk Dashboard and Tyk Gateway pods. I'm seeing the SSL error (see comments below) when trying to connect from Gateway to Dashboard (Dashboard to Gateway is fine). The error goes away if I rebuild everything without Istio. I must be doing something really silly. :( Both pods are in the same cluster, same namespace.
I managed to fix the issue. Redis wasn't the issue. Issue was communication from Tyk Gateway -> Tyk Dashboard was failing. The gateway talks to the dashboard to register its presence. The connection logs showed what looked like a tls origination issue with Istio envoy proxy when it is routing the traffic. I configured a DestinationRule that explicitly turned off mtls for the dashboard and the problem went away.
Scenario
Istio version 1.5.0 ontop of EKS 1.14.
Enabled components:
Base
Pilot
NOTE Istio 1.5.0 deprecates Mixer, moved to telemetry v2, which happens inside the envoy proxy sidecar.
I want to use Istio to support some metrics out of the box.
Here's the flow
my computer -> Gateway -> Virtual Service A -> Virtual Service B
I made sure that:
K8s Service objects have label app
K8s Deployment objects and their pod templates have label app
I can run the flow just fine, which means the configurations are correct.
The problem is with telemetry.
istio_requests_total{connection_security_policy="unknown",destination_app="unknown",destination_canonical_revision="latest",destination_canonical_service="unknown",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_service="svcb.default.svc.cluster.local",destination_service_name="svcb.default.svc.cluster.local",destination_service_namespace="unknown",destination_version="unknown",destination_workload="unknown",destination_workload_namespace="unknown",grpc_response_status="0",instance="10.2.55.80:15090",job="envoy-stats",namespace="default",pod_name="svca-77969dc86b-964p5",reporter="source",request_protocol="grpc",response_code="200",response_flags="-",source_app="svca",source_canonical_revision="latest",source_canonical_service="svca",source_principal="spiffe://cluster.local/ns/default/sa/default",source_version="unknown",source_workload="svca",source_workload_namespace="default"}
Question
Why are most destination-* labels unknown?
The official istio mesh dashboard typically filter metrics by reporter=destination. Why do all of my istio_requests_total series have reporter=source?
Oh right, after much digging, here's the answer.
Istio supports proxying all TCP traffic by default, but in order to provide additional capabilities, such as routing and rich metrics, the protocol must be determined. This can be done automatically or explicitly specified
I didn't specify the port name in my Service resource. Once I did that, the problem is resolved.
We can't get to Istio objects via Kubernetes REST API.
Example:
kubectl get gateways works and shows all Istio gateways in the default namespace.
curl ..../api/v1/namespaces/default/pods shows all the pods
deployed in the default name space.
curl ..../api/v1/namespaces/default/gateways returns 404.
Same is true for virtualservices,serviceentries, and any other Istio objects.
We have one REST API server running in the cluster. We are guessing the problem may be caused by it supporting API version v1 while Istio object creation YAML files refrence API version networking.istio.io/v1alpha3.
This is kinda confusing since we can create and get Istio objects via kubectl command but cannot do the same by issuing an HTTP request to kubernetes REST API server. Any insight would be welcome. Thanks.
I got a couple of ideas when I checked the logs of the kubernetes REST server. The rest server was discovered as a pod running in the namespace kube-system.
Anyhow, every time you need to use kubernetes REST server to get an Istio object created via API version networking.istio.io/v1alpha3 instead of issuing HTTP request to kubernetes REST server like so .../api/v1/namespaces/default/gateways do instead .../apis/networking.istio.io/v1alpha3/gateways. Replace gateways with the name of your Istio object of interest.