tl;dr
Can Istio do proxy resolution at mesh level? I want to be able to avoid defining HTTPS_PROXY in my service container.
Here is the setup:
My enterprise has an AKS cluster
In the AKS cluster we have Istio up and running.
Kubernetes Version 1.22.6
Istio Version 1.14.1
My service is running inside AKS and my pod has sidecar injection enabled.
SCENARIO:
My service needs to access two external services - www.google.com and secured.my-enterprise.com.
secured.my-enterprise.com is a service that is running in a private network on enterprise premises, NOT on Azure Cloud.
The only way to access secured.my-enterprise.com is to specify an HTTPS_PROXY which points to proxy.enterprise-proxy.svc.cluster.local.
This proxy (running inside AKS) knows how to get requests from AKS to on-premise infrastructure, navigating all the fancy network peering.
I dont however need the HTTPS_PROXY to access www.google.com.
Currently I am solving this problem the usual way - with NO_PROXY .google.com.
What I would like to be able to do:
Define a Virtual Service / Service Entry for host: www.google.com and egress out of AKS.
Define a Virtual Service / Service Entry for host: secured.my-enterprise.com, defining proxy.enterprise-proxy.svc.cluster.local as the HTTPS_PROXY to use for it, and egress out of AKS.
What would I achieve with this?
My service does not have to manage a NO_PROXY based on what it needs to do, and the service mesh can handle figuring out how to reach an external resource.
Is there something in the Istio toolset that can help me achieve this?
I have gone through Service Entry documentation and can't for the life of me understand how to add a proxy into the mix :)
Thanks!!
PS: Created a topic for this on discuss.istio.io as well
Related
I have some basic questions about Istio. I installed Istio for my Tyk API gateway. Then I found that simply installing Istio will cause all traffic between the Tyk pods to be blocked. Is this the default behaviour for Istio? The Tyk gateway cannot communicate with the Tyk dashboard.
When I rebuild my deployment without Istio, everything works fine.
I have also read that Istio can be configured with virtual services to perform traffic routing. Is this what I need to do for every default installing of Istio? Meaning, if I don't create any virtual services, then Istio will block all traffic by default?
Secondly, I understand a virtual service is created as a YAML file applied as a CRD. The host name defined in the virtual service rules - in a default Kubernetes cluster implementation on Google Cloud, how do I find out the host name of my application?
Lastly, if I install Tyk first, then later install Istio, and I have created the necessary label in Tyk's nanmespace for the proxy to be injected, can I just perform a rolling upgrade of my Tyk pods to have Istio start the injection?
For example, I have these labels in my Tyk dashboard service. Do I use the value called "app" in my virtual service YAML?
labels:
app: dashboard-svc-tyk-pro
app.kubernetes.io/managed-by: Helm
chart: tyk-pro-0.8.1
heritage: Helm
release: tyk-pro
Sorry for all the basic questions!
For question on Tyk gateway cannot communicate with the Tyk dashboard.
(I think the problem is that your pod tries to connect to the database before the Istio sidecar is ready. And thus the connection can't be established.
Istio runs an init container that configures the pods route table so all traffic is routed through the sidecar. So if the sidecar isn't running and the other pod tries to connect to the db, no connection can be established. Ex case: Application running in Kubernetes cron job does not connect to database in same Kubernetes cluster)
For question on Virtual Services
2.Each virtual service consists of a set of routing rules that are evaluated in order, letting Istio match each given request to the virtual service to a specific real destination within the mesh.
By default, Istio configures the Envoy proxies to passthrough requests to unknown services. However, you can’t use Istio features to control the traffic to destinations that aren’t registered in the mesh.
For question on hostname refer to this documentation.
The hosts field lists the virtual service’s hosts - in other words, the user-addressable destination or destinations that these routing rules apply to. This is the address or addresses the client uses when sending requests to the service.
Adding Istio on GKE to an existing cluster please refer to this documentation.
If you want to update a cluster with the add-on, you may need to first resize your cluster to ensure that you have enough resources for Istio. As when creating a new cluster, we suggest at least a 4 node cluster with the 2 vCPU machine type.If you have an existing application on the cluster, you can find out how to migrate it so it's managed by Istio as mentioned in the Istio documentation.
You can uninstall the add-on following document which includes to shift traffic away from the Istio ingress gateway.Please take a look at this doc for more details on installing and uninstalling Istio on GKE.
Also adding this document for installing Istio on GKE which also includes installing it to an existing cluster to quickly evaluate Istio.
I'm using lb for prod site. But for internal services (like gitlab, jenkins) I don't want to host a lb. Is there any alternative way to connect to internal services without the use of load balancers? Like could any bastion host do the job?
Having lb for prod and internal services seems to cost around 35 to 45 dollars. I'm trying to reduce the total bill.
I have a nginx ingress controller for production site, wondering if I could do something with it using subdomains for internal services.
Services on a Kubernetes cluster can talk to each other using their ClusterIP, without any use of a load balancer. The IP can be returned using the internal DNS service such as KubeDNS or CoreDNS
For more details
If you need to connect services in different clusters i’d take a look at Kubefed or Submarier for Multi-Cluster deployment.
Which by the way also harness the use of DNS service discovery (this is the term used for in-kubernetes service communication) for service-to-service communication.
Kubefed
Submariner
i am running a minimal stateful database service on GKE. single node cluster. i've setup a database as a stateful set on a single pod as of now. the database has exposed a management console on a particular port along with the mandatory database port. i am attempting to do two things.
expose management port over a global HTTP(S) load balancer
expose database port outside of GKE to be consumed by the likes of Cloud Functions or App Engine Applications.
My stateful set is running fine and i can see from the container logs that the database is properly booted up and is listening on required ports.
i am attempting to setup a standalone NEG (ref: https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg) using a simple ClusterIP service.
the cluster service comes up fine and i can see it using
kubectl get service service-name
but i dont see the NEG setup as such... the following command returns nothing
$ gcloud compute network-endpoint-groups list
Listed 0 items.
my pod exposes the port 8080 my service maps 51000 to 8080 and i have provided the neg annotation
cloud.google.com/neg: '{"exposed_ports": {"51000":{}}'
I dont see any errors as such but neither do i see a NEG created/listed.
Any suggestions on how i would go about debugging this.
As a followup question...
when exposing NEG over global load balancer, how do i enforce authn?
im ok with either of service account roles or oauth/openid.
would i be able to expose multiple ports using a single NEG? for
e.g. if i wanted to expose one port to my global load balancer and
another to local services, is this possible with a single NEG or
should i expose each port using a dedicated ClusterIP service?
where can i find documentation/specification for google kubernetes
annotations. i tried to expose two ports on the neg using the
following annotation syntax. is that even supported/meaningful?
cloud.google.com/neg: '{"exposed_ports": {"51000":{},"51010":{}}'
Thanks in advance!
In order to create the service that is backed by a network endpoint group, you need to be working on a GKE Cluster that is VPC Native:
https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg#before_you_begin
When you create a new cluster, this option is disabled by default and you must enable it upon creation. You can confirm if your cluster is VPC Native going to your Cluster details in GKE. It should appear like this:
VPC-native (alias IP) Enabled
If the cluster is not VPC Native, you won’t be able to use this feature as described on their restrictions:
https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips#restrictions
In case you have VPC Native enabled, make sure as well that the pods have the same labels “purpose:” and “topic:” to make sure they are members of the service:
kubectl get pods --show-labels
You can also create multi-port services as it is described on Kubernetes documentation:
https://kubernetes.io/docs/concepts/services-networking/service/#multi-port-services
I can access the kubernetes api to get the deployments using Kubernetes proxy.
I get the list of deployments with:
127.0.0.1:8001/apis/apps/v1/deployments
This is getting the deployments locally. But what should I use the HOST and PORT to access the deployments from the cluster not locally but using the aws server.
I am new to Kubernetes, if the question is not understandable please let me know.
Any help is appreciated.
kubectl proxy forwards your traffic localy adding your authentication for you
Your public api endpoint can be exposed in different ways (or it can be completely inaccessible from public network) depending on your cluster setup.
In most cases it would be exposed on something like ie. https://api.my.cluster.fqdn or with custom port like https://api.my.cluster.fqdn:6443 and it would require authentication by ie. getting a baerer token or using client certificate. It is reasonable to use some client library to connect to API.
I'm investigating a move to Kubernetes (coming from AWS ECS). But I haven't solved the local development issue when depending on internal services.
Let me elaborate:
When developing and testing microservices, before they are deployed as a Kubernetes Service I want to be able to talk to other, internal Kubernetes Services. As there are > 20 microservices I have a Kubernetes cluster running latest development versions. I can't run a MiniKube.
example:
I'm developing an user-service which needs access to the email service. The Email service is already on Kubernetes and is an internal service.
So before the user-service is deployed I want to be able to talk to the internal email service for dev/testing. I can't make use of K8S nice service discovery env vars.
As we currently already have a VPN up to restrict DEV env to testers/development only, could I use this VPN to provide access to the Kubernetes-Service IP-addresses? I do have Kubernetes DEV-env on the same VPC as the VPN is in.
If you deploy your internal services as type NodePort, then you can access them over your VPN via that nodePort. NodePorts can be dynamically allocated or you can customize them to be 'static' where they are known by you up front.
When developing an app on your local machine, you can access the dependent service by that NodePort.
As an alternative, you can use port-forwarding from kubectl (https://kubernetes.io/docs/user-guide/connecting-to-applications-port-forward/) to forward a pod to your local machine. (Note: This only handles traffic to a pod not a service).
Telepresence (http://telepresence.io) is designed for this scenario, though it presumes developers have kubectl access to the staging/dev cluster.