I have recently setup an EKS cluster with Fargate.
When I tried to deploy Redis Service on k8s using guide, I am getting the following errors:
Pod provisioning timed out (will retry) for pod: default/redis-operator-79d6d769dc-j246j
Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
For solving the above errors, I tried the following solutions but none of them worked
Created a NAT gateway for granting internet connection to the instances in the private subnets.
Updated CoreDNS to run pods on Fargate. Reference
The NAT gateway that I created was in the private subnet. The private subnets themselves don't have any access to the internet. Hence, I was stuck in a loop.
By creating a nat gateway in a public subnet and then adding in the router table of private subnets being used by the EKS cluster I was able to schedule the pods
Related
When deploying an EKS cluster, the best practice is to deploy the managed control plane in private subnets. In terms of accessibility, the defalt option is public cluster, meaning that I can access it locally with kubectl tool and updated kubeconfig.
How am I able to access the cluster if it is deployed in private subnets with no inbound traffic? As per the documentation, AWS creates a managed endpoint that can access the cluster from within the AWS network.
What is the architecture behind it, how does it internally work? Is there some kind of a proxy (agent) being deployed (found aws-node)?
deployed my own EKS cluster
read the documentation
tried to scrape for additional info
The type of EKS networking you're setting up is configured to restrict access to the API server with a private endpoint that's only accessible from within the VPC. So any Kubernetes API requests (kubectl commands) have to originate from within the VPC (public or private subnets). If you are doing this as a personal project, then you can do the following:
Create a bastion host in the public subnet of your VPC with a key pair. Launch this host with user data that installs kubectl and any other CLI tools you need.
Access the bastion host via SSH from your workstation to ensure it works as expected.
Check that the security group attached to your EKS control plane can receive 443 traffic from the public subnet. You can create a rule for this if one doesn't exist. This will enable communication between the bastion host in the public subnet and the cluster in the private subnets.
Access the bastion host and then use it to communicate with the cluster just as you would with your personal machine. For example, run aws eks --region <region> update-kubeconfig --name <name-of-your-cluster> to update your kubeconfig and then proceed to run kubectl commands.
Sidenote:
If this is for an enterprise project, you can also look into using AWS VPN or DirectConnect to access the VPC.
Other helpful resources:
https://aws.amazon.com/blogs/containers/de-mystifying-cluster-networking-for-amazon-eks-worker-nodes/
https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html#private-access
In this question, the author says that the gke cluster is not available from other subnets in the VPC.
BUT, that is exactly what I need to do. I've added detail below, all suggestions welcome.
I created a VPC in Google Cloud with custom sub-nets. I have a subnet in us-east1 and another in us-east4. Then, I created a VPC-native private GKE cluster in the same VPC in the us-east4 subnet.
[added details]
GKE in us-east4
endpoint 10.181.15.2
control plane 10.181.15.0/28
pod address range 10.16.0.0/16
service address range 10.17.0.0/22
VPC subnet in us-east4
10.181.11.0/24
VPC subnet in us-east1
10.171.1.0/24
I added 10.171.1.0/24 as a Control Plane authorized network, and I added 10.171.1.0/24 to the automatically created firewall rule.
But I still can't use kubectl from the instance in the 10.171.1.0/24 subnet.
What I see when trying to use kubectl from a VM in us-east4 10.181.11.7
On this VM, I set the context with kubectl config use-context <correct gke context> and I have gcloud configured correctly. Then,
kubectl get pods correctly gives a list of pods in the gke cluster.
from a VM in us-east4 10.171.1.0 subnet, which is set up in the same way, kubectl get pods times out with an error that it's unable to reach the endpoint. The message is:
kubectl get pods
Unable to connect to the server: dial tcp 10.181.15.2:443: i/o timeout
This seems like a firewall problem, but I've been unable to find a solution, despite the abundance of GKE documentation out there. It could be a routing problem, but I thought VPC-native GKE cluster would take care of the routes automatically?
By default, the private endpoint for the control plane is accessible from clients in the same region as the cluster. If you want clients in the same VPC but located in different regions to access the control plane, you'll need to enable global access using the --enable-master-global-access option. You can do this when you create the cluster or you can update an existing cluster.
I have an architecture similar to https://github.com/aws-samples/ecs-refarch-cloudformation
I would like to know if I can have an AutoScalingGroup with instances in private subnets without using NAT Gateway?
I was experimenting with removing the NATs and adding VPC endpoints but I always end up with problem like this:
2022-08-21 10:55:07 UTC+1000 <MY_ECS_TEMPLATE> CREATE_FAILED The following resource(s) failed to create: [ECSAutoScalingGroup].
2022-08-21 10:55:07 UTC+1000 ECSAutoScalingGroup CREATE_FAILED Received 0 SUCCESS signal(s) out of 1. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement
if I can have an AutoScalingGroup with instances in private subnets without using NAT Gateway?
Sadly no. But you can create a vpc interface endpoints for ecs which will enable communication with ecs from your private subnets, without the need for internet and nat.
But the error can be also from cloudformation (CFN) which can't get information back if ec2 instanses launched correctly. So you may need to add vpc endpoints for CFN as well.
I had similar issue. We use AWS Batch, which is based on AutoScaling Groups. We use squid proxy for internet access from private subnets. After I created few endpoints as mentioned before, spot VM started. Next I had to update AMI for these VM and set proper proxy/noproxy. And it works.
I am having a use case where I need to deploy the EKS cluster in private subnets and access it through API Gateway.
Currently, if I deploy the EKS cluster in public subnet and try to access it is working fine. However, it is not working when the EKS cluster is deployed into the private subnet.
Currently API gateway is configured with VPC link to access EKS cluster securely.
Network load balancer is configured to connect to the EKS cluster nodes.
Please let me know if there is anything that I am missing here.
Thanks,
Avinash
I created an private eks cluster by terraform, and I followed the guide at this page: https://tf-eks-workshop.workshop.aws/500_eks-terraform-workshop.html, and network architecture is below:
then I deployed my web application in the eks cluster, and the application only access aws resource, so it works as expected, but the problem is the eks is in private vpc, and internet alb can't attached to it, so I'm not able to access my application from public web browser, is there any idea to impliment it ?
I've setup the vpc peerring between eks vpc and cicd vpc.
For the application deploymetn part, I create a service, which type is NodePort
Unfortunately the guide link here doesn't open for me, it seems from the VPC Diagram that you have a private subnet within your VPC for EKS, you can do the following -
Create a Private Ingress resource Using ALB Ingress (For Routing traffic based on hostname within the services within the cluster) - https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html
And then route traffic from Route53 to Internal LB, if its not reachable still, attach a LB in the default VPC and pass it on to the Internal Ingress Controller. Logically that should work.
EC2s in the same VPC can talk to each other if SGs are open, so by that logic forwarding traffic from Public Subnet to Private shouldn't be an issue.