I built a new cluster with Terraform for a AWS EKS, single node group with a single node.
This cluster is using 1.22 and cant seem to get anything to work correctly.
So Istio will install fine, i have installed versions 1.12.1, 1.13.2, 1.13.3 & 1.13.4 and all seem to have the same issue with auto injecting the sidecar.
Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded
But there are also other issues with the cluster, even without using Istio. My application is pulled and the pod will build fine but can not connect to the database. This is an external DB to the cluster - no other build (running on Azure) have any issues connecting to the DB
I am not sure if this is the issue with the application not connecting to the ext. DB but the sidecar issue could have something to do with BoundServiceAccountTokenVolume?
There is a warming about it being enabled on all clusters from 1.21 - a little odd as i have another applications with istio, running on another cluster with 1.21 on AWS EKS!
I also have this application running with istio without any issues in Azure on 1.22
I seem to have fix it :)
It seems to be a port issue with the security groups. I was letting terraform build its own group.
When I opened all the ports up in the 'inbound' section it seemed to work.
I then closed them all again and only opened 80 and 443 - which again stopped Istio from auto-injecting its sidecar
My app was requesting to talk to Istio on port 15017, so i opened just that port, along sided ports 80 and 443.
Once that port was opened, my app started to work and got the sidecar from Istio without any issue.
So it seems like the security group stops pod-to-pod communication... unless i have completely messed up my terraform build in some way
Related
In kubernetes (AWS EKS) v1.20 have a default namespace with two pods, connected with a service type loadbalancer (CLB). Requesting the uri to the CLB worked fine and routed to either of the pods as required.
Post installation of 1.13.0 of Istio with istio-injection=enabled label set on a different namespace, the communication of the non-istio pods with no sidecar injection doesnt seem to work.
What I mean by doesnt work: (below 3 scenarios always worked without istio)
curl requests sent to https://default-nspods/apicall always worked with the non-istio pods.
i.e., the CLB always forwarded requests to to the 2 pods as required.
curl request after logging into the pod1 to pod2s IP worked and vice versa.
curl request to pod2 uri from the Node1 of pod1 worked and vice versa.
Post
Post installation, 2 and 3 doesnt work. The CLB also has trouble reading the nodeport of the pods at times.
Ive checked istioctl proxy-config endpoints and checked the deployments where the sidecar injection is enabled, the output doesnt show any other non mesh service/pod details.
Istio Version: 1.13.0
Ingress Gateway: Enabled (Loadbalancer mode)
Egress Gateway: Disabled
No addons like Kiali, Prometheus
Istio Operator based installation with modified yaml values.
Single cluster installation i.e., ISTIO_MESH_ROUTER_MODE='standard'
Istio pods, envoy sidecars, proxy-config dont show any errors.
Am kind of stuck, please let me know if I need to check kube-proxy, ip-tables or some where else.
Ive uninstalled istio using the "istioctl x uninstall --purge" option and re-installed , but the non-mesh pods seem to be not working now with Istio installed or not.
Istio pods and Istio injection namespace pods dont have issues.
I have some basic questions about Istio. I installed Istio for my Tyk API gateway. Then I found that simply installing Istio will cause all traffic between the Tyk pods to be blocked. Is this the default behaviour for Istio? The Tyk gateway cannot communicate with the Tyk dashboard.
When I rebuild my deployment without Istio, everything works fine.
I have also read that Istio can be configured with virtual services to perform traffic routing. Is this what I need to do for every default installing of Istio? Meaning, if I don't create any virtual services, then Istio will block all traffic by default?
Secondly, I understand a virtual service is created as a YAML file applied as a CRD. The host name defined in the virtual service rules - in a default Kubernetes cluster implementation on Google Cloud, how do I find out the host name of my application?
Lastly, if I install Tyk first, then later install Istio, and I have created the necessary label in Tyk's nanmespace for the proxy to be injected, can I just perform a rolling upgrade of my Tyk pods to have Istio start the injection?
For example, I have these labels in my Tyk dashboard service. Do I use the value called "app" in my virtual service YAML?
labels:
app: dashboard-svc-tyk-pro
app.kubernetes.io/managed-by: Helm
chart: tyk-pro-0.8.1
heritage: Helm
release: tyk-pro
Sorry for all the basic questions!
For question on Tyk gateway cannot communicate with the Tyk dashboard.
(I think the problem is that your pod tries to connect to the database before the Istio sidecar is ready. And thus the connection can't be established.
Istio runs an init container that configures the pods route table so all traffic is routed through the sidecar. So if the sidecar isn't running and the other pod tries to connect to the db, no connection can be established. Ex case: Application running in Kubernetes cron job does not connect to database in same Kubernetes cluster)
For question on Virtual Services
2.Each virtual service consists of a set of routing rules that are evaluated in order, letting Istio match each given request to the virtual service to a specific real destination within the mesh.
By default, Istio configures the Envoy proxies to passthrough requests to unknown services. However, you can’t use Istio features to control the traffic to destinations that aren’t registered in the mesh.
For question on hostname refer to this documentation.
The hosts field lists the virtual service’s hosts - in other words, the user-addressable destination or destinations that these routing rules apply to. This is the address or addresses the client uses when sending requests to the service.
Adding Istio on GKE to an existing cluster please refer to this documentation.
If you want to update a cluster with the add-on, you may need to first resize your cluster to ensure that you have enough resources for Istio. As when creating a new cluster, we suggest at least a 4 node cluster with the 2 vCPU machine type.If you have an existing application on the cluster, you can find out how to migrate it so it's managed by Istio as mentioned in the Istio documentation.
You can uninstall the add-on following document which includes to shift traffic away from the Istio ingress gateway.Please take a look at this doc for more details on installing and uninstalling Istio on GKE.
Also adding this document for installing Istio on GKE which also includes installing it to an existing cluster to quickly evaluate Istio.
I'm trying to create kubernetes cluster, but whatever mode I try in the end kubectl fails with
The connection to the server ASFASF.da2.ap-northeast-1.eks.amazonaws.com was refused - did you specify the right host or port?
I already tried:
1) Install with Terraform ( following all official docs )
2) Manual EKS installation through UI
3) Install with eksctl tool
It all ends with this error, I already tried to tweak all possible subnets, roles, users, routes, dns, basically everything what might help in other SA threads / github issues, but no success..
What am I missing?
inb4 yes I tried update kubectl config or specify role there
like here:
https://github.com/weaveworks/eksctl/issues/1510
https://medium.com/#savvythrough/aws-eks-auth-optimization-for-k8s-ae054be0a31b
vpn, turning it on might help
I spent lot of time on this issue and it all because some DNS blocks in the area I was connecting from, lol
I am getting the following error while creating a gateway for the sample bookinfo application
Internal error occurred: failed calling admission webhook
"pilot.validation.istio.io": Post
https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s:
Address is not allowed
I have created a EKS poc cluster using two node-groups (each with two instances), one with t2.medium and another one is with t2.large type of instances in my dev AWS account using two subnets with /26 subnet with default VPC-CNI provided by EKS
But as the cluster is growing with multiple services running, I started facing issues of IPs not available (as per docs default vpc-cni driver treat pods as an EC2 instance)
to avoid same I followed following post to change networking from default to weave
https://medium.com/codeops/installing-weave-cni-on-aws-eks-51c2e6b7abc8
because of same I have resolved IPs unavailability issue,
Now after network reconfiguration from vpc-cni to weave
I am started getting above issue as per subject line for my service mesh configured using Istio
There are a couple of services running inside the mesh and also integrated kiali, prometheus, jaeger with the same.
I tried to have a look at Github (https://github.com/istio/istio/issues/9998) and docs
(https://istio.io/docs/ops/setup/validation/), but could not get a proper valid answer.
Let me if anyone face this issue and have partial/full solution on this.
This 'appears' to be related to the switch from AWS CNI to weave. CNI uses the IP range of your VPC while weave uses its own address range (for pods), so there may be remaining iptables rules from AWS CNI, for example.
Internal error occurred: failed calling admission webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Address is not allowed
The message above implies that whatever address istio-galley.istio-system.svc resolves to, internally in your K8s cluster, is not a valid IP address. So I would also try to see what that resolves to. (It may be related to coreDNS).
You can also try the following these steps;
Basically, (quoted)
kubectl delete ds aws-node -n kube-system
delete /etc/cni/net.d/10-aws.conflist on each of the node
edit instance security group to allow UDP, TCP on 6873, 6874 ports
flush iptables, nat, mangle, filter
restart kube-proxy pods
apply weave-net daemonset
delete existing pods so the get recreated in Weave pod CIDR's address-space.
Furthermore, you can try reinstalling everything from the beginning using weave.
Hope it helps!
Using AWS to run kubernetes cluster which installed by kops.
Using alb-ingress-controller to realize application load balancer(ALB) on AWS.
Deployed a websocket application into the kubernetes cluster. When try to access the application from ALB DNS, because of it's using load balancer, so sometimes can't catch the response. It's went to the target group one by one.
Maybe the default proxy-mode of kube-proxy is iptables, so thinking about change iptable to userspace, here is a related question:
What does userspace mode means in kube-proxy's proxy mode?
But is it the right way? To make sure the websocket application can run correctly on AWS using ALB?
If it's the right way, then how to change kube-proxy's proxy-mode to userspace?
Here is another article about running socket.io application on kubernetes:
Running Socket.IO Applications on Kubernetes
But it's using ELB. And the method seems a little complex.
Finding a good way.