In this question, the author says that the gke cluster is not available from other subnets in the VPC.
BUT, that is exactly what I need to do. I've added detail below, all suggestions welcome.
I created a VPC in Google Cloud with custom sub-nets. I have a subnet in us-east1 and another in us-east4. Then, I created a VPC-native private GKE cluster in the same VPC in the us-east4 subnet.
[added details]
GKE in us-east4
endpoint 10.181.15.2
control plane 10.181.15.0/28
pod address range 10.16.0.0/16
service address range 10.17.0.0/22
VPC subnet in us-east4
10.181.11.0/24
VPC subnet in us-east1
10.171.1.0/24
I added 10.171.1.0/24 as a Control Plane authorized network, and I added 10.171.1.0/24 to the automatically created firewall rule.
But I still can't use kubectl from the instance in the 10.171.1.0/24 subnet.
What I see when trying to use kubectl from a VM in us-east4 10.181.11.7
On this VM, I set the context with kubectl config use-context <correct gke context> and I have gcloud configured correctly. Then,
kubectl get pods correctly gives a list of pods in the gke cluster.
from a VM in us-east4 10.171.1.0 subnet, which is set up in the same way, kubectl get pods times out with an error that it's unable to reach the endpoint. The message is:
kubectl get pods
Unable to connect to the server: dial tcp 10.181.15.2:443: i/o timeout
This seems like a firewall problem, but I've been unable to find a solution, despite the abundance of GKE documentation out there. It could be a routing problem, but I thought VPC-native GKE cluster would take care of the routes automatically?
By default, the private endpoint for the control plane is accessible from clients in the same region as the cluster. If you want clients in the same VPC but located in different regions to access the control plane, you'll need to enable global access using the --enable-master-global-access option. You can do this when you create the cluster or you can update an existing cluster.
Related
When deploying an EKS cluster, the best practice is to deploy the managed control plane in private subnets. In terms of accessibility, the defalt option is public cluster, meaning that I can access it locally with kubectl tool and updated kubeconfig.
How am I able to access the cluster if it is deployed in private subnets with no inbound traffic? As per the documentation, AWS creates a managed endpoint that can access the cluster from within the AWS network.
What is the architecture behind it, how does it internally work? Is there some kind of a proxy (agent) being deployed (found aws-node)?
deployed my own EKS cluster
read the documentation
tried to scrape for additional info
The type of EKS networking you're setting up is configured to restrict access to the API server with a private endpoint that's only accessible from within the VPC. So any Kubernetes API requests (kubectl commands) have to originate from within the VPC (public or private subnets). If you are doing this as a personal project, then you can do the following:
Create a bastion host in the public subnet of your VPC with a key pair. Launch this host with user data that installs kubectl and any other CLI tools you need.
Access the bastion host via SSH from your workstation to ensure it works as expected.
Check that the security group attached to your EKS control plane can receive 443 traffic from the public subnet. You can create a rule for this if one doesn't exist. This will enable communication between the bastion host in the public subnet and the cluster in the private subnets.
Access the bastion host and then use it to communicate with the cluster just as you would with your personal machine. For example, run aws eks --region <region> update-kubeconfig --name <name-of-your-cluster> to update your kubeconfig and then proceed to run kubectl commands.
Sidenote:
If this is for an enterprise project, you can also look into using AWS VPN or DirectConnect to access the VPC.
Other helpful resources:
https://aws.amazon.com/blogs/containers/de-mystifying-cluster-networking-for-amazon-eks-worker-nodes/
https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html#private-access
I have recently setup an EKS cluster with Fargate.
When I tried to deploy Redis Service on k8s using guide, I am getting the following errors:
Pod provisioning timed out (will retry) for pod: default/redis-operator-79d6d769dc-j246j
Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
For solving the above errors, I tried the following solutions but none of them worked
Created a NAT gateway for granting internet connection to the instances in the private subnets.
Updated CoreDNS to run pods on Fargate. Reference
The NAT gateway that I created was in the private subnet. The private subnets themselves don't have any access to the internet. Hence, I was stuck in a loop.
By creating a nat gateway in a public subnet and then adding in the router table of private subnets being used by the EKS cluster I was able to schedule the pods
I am trying to access the kubectl of the master node that is running on an ec2 instance. I want to do this from another ec2 instance running on a different vpc. What steps should I take to make this possible?
I have the kubeconfig file on my second machine already but on running kubectl, it gives me a connection error,
Edit: Both the vpcs are private and have the similar CIDR.
If both of your EC2 are in diff VPCs you can do the VPC peering.
If you want to expose your master and K8s setup you can directly use the public IP(if exist) of EC2 and kubectl will connect to k8s cluster over the internet.
You can also checkout peering multiple VPC with same cidr range if you are looking for that way : Multiple VPC and Subnet with same CIDR blocks
Or : https://docs.aws.amazon.com/vpc/latest/peering/peering-configurations-partial-access.html#two-vpcs-peered-specific-cidr
If your eks api server is private, create peering between the
VPCs and allow your Second EC2 server's private IP
If your eks api server is public, you can allow your Second EC2 instance's public IP from the aws console, in the eks security or network section
I mistakenly deleted the firewall entry that allows the control plane nodes to establish ssh tunnels to the worker nodes. I need to recreate the firewall entry, but I can't find the IP addresses of the controller nodes.
This is a regional cluster, so the cluster endpoint is a load balancer that sits in front of the 3 control plane IPs. I don't see that load balancer in my GCP console though, so I can't get its details.
How do I find the IP addresses of the 3 control plane nodes in a GKE regional cluster?
There should have been a log created when the rules were deleted. The IP addresses are in there. You can pull them out with something like this:
gcloud logging read 'resource.type="gce_firewall_rule" AND \
timestamp>="2020-11-01T00:00:00Z" AND \
protoPayload.methodName="v1.compute.firewalls.delete" AND \
protoPayload.resourceName:gke'
As per the doc you can see the control plane CIDR block of GKE cluster. Use below gcloud command:
gcloud container clusters describe cluster-name --zone cluster-zone
As per the doc if you delete or modify the firewall rules created by GKE you might encounter unexpected behavior in your clusters.
Also as per this GCP document masterIpv4CidrBlock is deprecated.
we can find the private ip of control plane of GKE cluster using below command
gcloud container clusters describe cluster-name --zone zone-name --format="value(privateClusterConfig)"
for eg : gcloud container clusters describe private-cluster-0 --zone us-east1-b --format="value(privateClusterConfig)"
I have limited IP's in my public facing VPC which basically means I cannot run the K8S worker nodes in that VPC since I would not have sufficient IP's to support all the pods. My requirement is to run the control plane in my public facing VPC and the worker nodes in a different VPC with a private IP range (192.168.X.X).
We use traefik for ingress and have deployed traefik as a DaemonSet. These pods are exposed using a Kubernetes service of type NLB. And we created a VPC endpoint on top of this NLB which allows us to access this traefik endpoint through our public facing VPC.
However, based on docs it looks like NLB support is still in alpha stage. I am curious what are my other options given the above constraints.
Usually, in Kubernetes cluster Pods are running in separate overlay subnet that should not overlap with existing IP subnets in VPC.
This functionality is provided by Kubernetes cluster networking solutions like Calico, Flannel, Weave, etc.
So, you only need to have enough IP address space to support all cluster nodes.
The main benefit of using NLB is to expose client IP address to pods, so if there are no such requirements, regular ELB would be good for most cases.
You can add secondary CIDR to your vpc and use one of the two options mentioned here to have pods use the secondary vpc CIDR.
https://aws.amazon.com/blogs/containers/optimize-ip-addresses-usage-by-pods-in-your-amazon-eks-cluster/