I've created
1 GKE autopilot cluster
1 classic VPN Tunnel to on-premise network
Network Connectivity tests suggest, that my VPN is working (I suppose)
Result of Connectivity Test
When trying to traceroute from a pod however:
traceroute from pod
Pod IP is 10.4.0.85
I have a host project (network) which also contains VPN Tunnel and routing. This network is shared with another project, which contains the GKE autopilot cluster.
VPN tunnel show to be working from both ends. GKE Nodes are pingable from on-premise network.
Since it is a autopilot cluster, I cannot confirm, whether connection from nodes to on-premise is working.
I expected my traceroute to show successful connection to on premise IP, or at least to VPN Endpoint of on-premise network.
It only shows one hop to 10.4.0.65 (this is in the CIDR of my GKE Cluster, but I do not know where it belongs to)
I've taken a look at the IP masquerading as described here without success.
And now I am lost. I suppose, my packages (from traceroute) are not even leaving the GKE cluster. But why that is, I cannot tell.
Id be grateful to get a hint in the right direction.
Related
i have a cluster of 2 nodes created in gcp. the worker node (L1 VM) has nested virtualization enabled. i have created a pod in this L1 VM. and i have launched a L2 VM using qemu in this pod.
my objective is to access this L2 VM only by a IP address from external word (internet). there are many services running in my VM (L2 VM) and i need to access it only by IP.
i created a tunnel from node to L2 VM (which is within pod) to get dhcp address to my VM. but it seems dhcp offer and ack messages are blocked by google cloud.
i have got a public IP in the cluster through which i can reach to private IP of node. most probably there is a NAT configured in the cloud for the node's private IP.
can i configure node as a NAT gw so that i can push this packet further from internet to L2 VM?
any other suggestions are welcome!
I think you are trying to implement something like a bastion host. However, this is something that you shouldn't do with kubernetes. Although you 'can' implement it with kubernetes, it is simply not made for it.
I see there two vivid options for you:
A. Create another virtual machine (GCE instance) inside the same VPC as the cluster and set it up as a bastion host or as an endpoint for a VPN.
B. You can use the identity aware proxy (IAP) to tunnel the traffic to your machine inside the VPC as described here
The IAP is probably the best solution for your usecase.
Also consider using simple GCE instances as opposed to a kubernetes cluster. A kubernetes cluster is very useful if you have a lot of workload that is too much for one node or if you need to scale out and in etc. Your usecase looks to me more that you still think in the traditional server world and less about cattle vs pets.
My question here is maybe simple but I'm missing something.
I have a GKE cluster and I want to connect it to an on-premise database.
GKE - VPC-native cluster
GKE version - 1.17.15-gke.800
POD ip range - 10.0.0.0/14
SERVICES ip range - 10.4.0.0/20
I have a cloud VPN working (policy based connection) and I have a connection from Google's network to the onpremise network. I've tested it from a test instance and from the instances of the GKE cluster.
I don't have connection only from the pods.
What am I missing here ?
I managed to find the right answer:
Egress traffic from GKE Pod through VPN
Got it from here, I needed to enable Network Policy for master + nodes and then used the ip-masq-agent config to create a Configmap, then you must delete the pods of ip-masq-agent and when they come up with the new config everything is working fine.
My co-workers are launching GKE clusters and managing them from a pair of centralized VMs. The vms are in us-east4
When they launch GKE clusters in the same region (us-east4), all is well. They can access both the worker nodes and also the GKE Master addresses via the peering connection. However, they could not access the master nodes of a GKE cluster built in europe-west3. I built a VM in that region, and was successfully able to connect to port 443 of the master node IPs. Global routing is enabled for the VPC network and inter-region access of VMs and other services is no problem.
Seems very clear that GKE master nodes can only be accessed in the same region. But is this documented somewhere? I did open a support case on Monday, but having little luck getting any reasonable information back.
It seems like this is an expected behavior. Since I have reviewed here, I understood the following information about it, but you are right, there is nothing like this on it:
The private IP address of the master in a regional cluster only could be reachable from the subnetworks that are in the same region, or from on-premises devices that are connected to the same region.
Now, based on this, I would recommend you to set up a proxy on the same region where your GKE master is, in order to make all the requests coming from a different region, look like they come from the reachable region.
Please review this, it is an example about how to reach your master from a cluster in another region.
Assuming I have a custom VPC with IP ranges 10.148.0.0/20
This custom VPC has firewall rules to allow-internal so the service inside those IP ranges can communicate to each other.
After the system grows I need to connect to some on-premises network by using Classic Cloud VPN, already create Cloud VPN (the on-premises side configuration already configured by someone) and the VPN Tunnel already established (with green checkmarks).
I also can ping to on-premises IP right now (let's say ping to 10.xxx.xxx.xxx where this is not GCP internal/private IP but on-premises private IP) using compute engine created on custom VPC network.
The problem is all the compute engine instance spawn in custom VPC network can't communicate to the internet now (like doing sudo apt update) or even communicate to google cloud storage (using gsutil), but they can communicate using private IP.
I also can't spawn dataproc cluster on that custom VPC (I guess because it can't connect to GCS, since dataproc needs GCS for staging buckets).
Since I do not really know about networking stuff and relatively new to GCP, how to be able to connect to the internet on instances that I created inside custom VPC?
After checking more in-depth about my custom VPC and Cloud VPN I realize there's misconfiguration when I establish the Cloud VPN, I've chosen route-based in routing option and input 0.0.0.0/0 in Remote network IP ranges. I guess this routes sending all traffic to VPN as #John Hanley said.
Solved it by using policy-based in routing option and only add specific IP in Remote network IP ranges.
Thank you #John Hanley and
#guillaume blaquiere for pointing this out
I'd like to setup a NAT gateway, using Cloud NAT, so that VMs/Pods in a public GKE cluster use static IP addresses.
The issue I'm facing is that the NAT gateway seems to only be used if VMs have no other options, i.e:
GCP forwards traffic using Cloud NAT only when there are no other matching routes or paths for the traffic.
But in the case of a public GKE cluster, VMs have ephemeral external IPs, so they don't use the gateway.
According to the doc:
If you configure an external IP on a VM's interface [...] NAT will not be performed on such packets. However, alias IP ranges assigned to the interface can still use NAT because they cannot use the external IP to reach the Internet.
And
With this configuration, you can connect directly to a GKE VM via SSH, and yet have the GKE pods/containers use Cloud NAT to reach the Internet.
That's what I want, but I fail to see what precisely to setup here.
What is implied by alias IP ranges assigned to the interface can still use NAT and how to set this up?
"Unfortunately, this is not currently the case. While Cloud NAT is still in Beta, certain settings are not fully in place and thus the pods are still using SNAT even with IP aliasing. Because of the SNAT to the node's IP, the pods will not use Cloud NAT."
Indeed, as Patrick W says above, it's not currently working as documented. I tried as well, and spoke with folks on the GCP Slack group in the Kubernetes Engine channel. They also confirmed in testing that it only works with a GKE private cluster. We haven't started playing with private clusters yet. I can't find solid documentation on this simple question: If I create a private cluster, can I still have public K8S services (aka load balancers) in that cluster? All of the docs about private GKE clusters indicate you don't want any external traffic coming in, but we're running production Internet-facing services on our GKE clusters.
I filed a ticket with GCP support about the Cloud NAT issue, and here's what they said:
"I have been reviewing your configuration and the reason that Cloud NAT is not working is because your cluster is not private.
To use Cloud NAT with GKE you have to create a private cluster. In the non-private cluster the public IP addresses of the cluster are used for communication between the master and the nodes. That’s why GKE is not taking into consideration the Cloud NAT configuration you have.
Creating a private cluster will allow you to combine Cloud NAT and GKE.
I understand this is not very clear from our documentation and I have reported this to be clarified and explained exactly how it is supposed to work."
I responded asking them to please make it work as documented, rather than changing their documentation. I'm waiting for an update from them...
Using google's Cloud NAT with public GKE clusters works!
First a cloud NAT gateway and router needs to be setup using a reserved external IP.
Once that's done the ip-masq-agent configuration needs to be changed to not masquerade the pod IPs for the external IPs that are the target of requests from inside the cluster. Changing the configuration is done in the nonMasqueradeCidrs list in the ConfigMap for the ip-masq-agent.
The way this works is that for every outgoing requests to an IP in the nonMasqueradeCidrs list IP masquerading is not done. So the requests does not seem to originate from the node IP but from the pod IP. This internal IP is then automatically NATed by the Cloud NAT gateway/router. The result is that the request seems to originate from the (stable) IP of the cloud NAT router.
Sources:
https://rajathithanrajasekar.medium.com/google-cloud-public-gke-clusters-egress-traffic-via-cloud-nat-for-ip-whitelisting-7fdc5656284a
https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent
The idea here is that if you use native VPC (Ip alias) for the cluster, your pods will not use SNAT when routing out of the cluster. With no SNAT, the pods will not use the node's external IP and thus should use the Cloud NAT.
Unfortunately, this is not currently the case. While Cloud NAT is still in Beta, certain settings are not fully in place and thus the pods are still using SNAT even with IP aliasing. Because of the SNAT to the node's IP, the pods will not use Cloud NAT.
This being said, why not use a private cluster? It's more secure and will work with Cloud NAT. You can't SSH directly into a node, but A) you can create a bastion VM instance in your project that can SSH using the internal IP flag and B) you generally do not need to SSH into the node on most occassions.