I am experiencing an issue on my EKS cluster (kubernetes version 1.14).
I am trying to install JupyterHub (version 0.8.2) onto the EKS cluster, via helm+tiller.
Install appears to succeed, but the proxy-public service fails to create an ELB. See the output of "kubectl describe svc" below:
The app install/deployment fails with an error event.
> kubectl describe svc
...
Name: proxy-public
Namespace: jhub
Labels: app=jupyterhub
chart=jupyterhub-0.8.2
component=proxy-public
heritage=Tiller
release=jhub
...
Warning CreatingLoadBalancerFailed 1m (x6 over 3m) service-controller Error creating load balancer (will retry): failed to ensure load balancer for service jhub/proxy-public: could not find any suitable subnets for creating the ELB
My EKS cluster is associated with 3 subnets, 2 private and 1 public. I would think that the ELB could be placed in the public subnet?
EKS requires subnets to be tagged in order to be used for load balancer creation. To be considered an eligible subnet, tag it with the following: kubernetes.io/role/elb: shared. For more info, see the knowledge portal article on subnet tagging for EKS.
Related
In this question, the author says that the gke cluster is not available from other subnets in the VPC.
BUT, that is exactly what I need to do. I've added detail below, all suggestions welcome.
I created a VPC in Google Cloud with custom sub-nets. I have a subnet in us-east1 and another in us-east4. Then, I created a VPC-native private GKE cluster in the same VPC in the us-east4 subnet.
[added details]
GKE in us-east4
endpoint 10.181.15.2
control plane 10.181.15.0/28
pod address range 10.16.0.0/16
service address range 10.17.0.0/22
VPC subnet in us-east4
10.181.11.0/24
VPC subnet in us-east1
10.171.1.0/24
I added 10.171.1.0/24 as a Control Plane authorized network, and I added 10.171.1.0/24 to the automatically created firewall rule.
But I still can't use kubectl from the instance in the 10.171.1.0/24 subnet.
What I see when trying to use kubectl from a VM in us-east4 10.181.11.7
On this VM, I set the context with kubectl config use-context <correct gke context> and I have gcloud configured correctly. Then,
kubectl get pods correctly gives a list of pods in the gke cluster.
from a VM in us-east4 10.171.1.0 subnet, which is set up in the same way, kubectl get pods times out with an error that it's unable to reach the endpoint. The message is:
kubectl get pods
Unable to connect to the server: dial tcp 10.181.15.2:443: i/o timeout
This seems like a firewall problem, but I've been unable to find a solution, despite the abundance of GKE documentation out there. It could be a routing problem, but I thought VPC-native GKE cluster would take care of the routes automatically?
By default, the private endpoint for the control plane is accessible from clients in the same region as the cluster. If you want clients in the same VPC but located in different regions to access the control plane, you'll need to enable global access using the --enable-master-global-access option. You can do this when you create the cluster or you can update an existing cluster.
I have recently setup an EKS cluster with Fargate.
When I tried to deploy Redis Service on k8s using guide, I am getting the following errors:
Pod provisioning timed out (will retry) for pod: default/redis-operator-79d6d769dc-j246j
Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
For solving the above errors, I tried the following solutions but none of them worked
Created a NAT gateway for granting internet connection to the instances in the private subnets.
Updated CoreDNS to run pods on Fargate. Reference
The NAT gateway that I created was in the private subnet. The private subnets themselves don't have any access to the internet. Hence, I was stuck in a loop.
By creating a nat gateway in a public subnet and then adding in the router table of private subnets being used by the EKS cluster I was able to schedule the pods
I have running EKS Cluster in AWS. Now I want to try use my own Network Load Balancer which were created not with AWS EKS annotations.
So my question: Is it even possible to use own NLB with EKS? If yes how can I do it? If not why it is not possible?
I've researched a lot found one opensource kind for EKS named as TargetGroupBinding, I've provided an ARN of my target group however than health checks are failing.
For some providers it is possible, such as Tencent Kubernetes Engine per official documentation -
metadata:
name: my-service
annotations:
# ID of an existing load balancer
service.kubernetes.io/tke-existed-lbid：lb-6swtxxxx
However this is not supported for AWS, as in the above link you can see that there is no similar annotation for AWS load balancers.
I am trying to expose a service to the outside world using the loadBalancer type service.
For that, i have followed this doc
https://aws.amazon.com/premiumsupport/knowledge-center/eks-kubernetes-services-cluster/
My loadbalancer.yaml looks like this
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
type: LoadBalancer
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
But the load balancer is not creating as expected I am getting the following error
Warning SyncLoadBalancerFailed 8s (x3 over 23s) service-controller Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB
Seems like its because of some issues in the subnet tags to solve,but i have the required tags in my subnets
kubernetes.io/cluster/<cluster-name>. owned
kubernetes.io/role/elb 1
But still, I am getting the error could not find any suitable subnets for creating the ELB
By default AWS EKS only attaches load balancers to public subnets. In order to launch it in a private subnet you need to not only label your subnets (which it looks like you did) but also annotate your load balancer-
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
You can find more information here.
For people that may reach this question, I have faced the same error, but the problem was really simple.
The tag with key kubernetes.io/cluster/<cluster-name> had the wrong cluster name as the automation that deployed it was wrong.
In EKS 1.16, I need internet-facing NLB.
The root cause in EKS is that you haven't selected a public subnet while creating the cluster.
After creating the cluster EKS will not allow to update subnets as of now here
To resolve the issue, I have performed the below steps
created a public subnet in the same vpc of EKS
Attached IGW in route tables in new created public subnets
Added below tags in public subnets
kubernetes.io/cluster/<EKSClusterName> : shared
Note: In a 4th step, Replace your EKS cluster name in placeholder EKSClusterName
Resolution This has resolved my issue.
To identify a cluster's subnets, the Kubernetes Cloud Controller
Manager (cloud-controller-manager) and AWS Load Balancer Controller
(aws-load-balancer-controller) query that cluster's subnets by using
the following tag as a filter:
Choose the appropriate option for tagging your subnets:
For public and private subnets used by load balancer resources
Tag all public and private subnets that your cluster uses for load balancer resources with the following key-value pair:
Key: kubernetes.io/cluster/cluster-name Value: shared
The cluster-name value is for your Amazon EKS cluster. The shared value allows more than one cluster to use the subnet.
For private subnets used by internal load balancers
To allow Kubernetes to use your private subnets for internal load balancers, tag all private subnets in your VPC with the following key-value pair:
Key: kubernetes.io/role/internal-elb Value: 1
For public subnets used by external load balancers
To allow Kubernetes to use only tagged subnets for external load balancers, tag all public subnets in your VPC with the following key-value pair:
Key: kubernetes.io/role/elb Value: 1
Note: Use the preceding tag instead of using a public subnet in each Availability Zone.
reference: https://aws.amazon.com/premiumsupport/knowledge-center/eks-vpc-subnet-discovery/
Possibly your subnet is not a public one, i.e. accessible from the internet. This will be required for your Loadbalancer to accept traffic from the outside world. In order to make it public, you need to attach an Internet Gateway to your VPC. Check here for more documentation.
Additional to Robert' answer, you can use the following kubectl command for annotating a service;
kubectl annotate svc <service-name> service.beta.kubernetes.io/aws-load-balancer-internal="true"
I'm trying to setup kubernetes on AWS. For this I created an EKS cluster with 3 nodes (t2.small) according to official AWS tutorial. Then I want to run a pod with some app which communicates with Postgres (RDS in different VPC).
But unfortunately the app doesn't connect to the database.
What I have:
EKS cluster with its own VPC (CIDR: 192.168.0.0/16)
RDS (Postgres) with its own VPC (CIDR: 172.30.0.0/16)
Peering connection initiated from the RDS VPC to the EKS VPC
Route table for 3 public subnets of EKS cluster is updated: route with destination 172.30.0.0/16 and target — peer connection from the step #3 is added.
Route table for the RDS is updated: route with destination 192.168.0.0/16 and target — peer connection from the step #3 is added.
The RDS security group is updated, new inbound rule is added: all traffic from 192.168.0.0/16 is allowed
After all these steps I execute kubectl command:
kubectl exec -it my-pod-app-6vkgm nslookup rds-vpc.unique_id.us-east-1.rds.amazonaws.com
nslookup: can't resolve '(null)': Name does not resolve
Name: rds-vpc.unique_id.us-east-1.rds.amazonaws.com
Address 1: 52.0.109.113 ec2-52-0-109-113.compute-1.amazonaws.com
Then I connect to one of the 3 nodes and execute a command:
getent hosts rds-vpc.unique_id.us-east-1.rds.amazonaws.com
52.0.109.113 ec2-52-0-109-113.compute-1.amazonaws.com rds-vpc.unique_id.us-east-1.rds.amazonaws.com
What I missed in EKS setup in order to have access from pods to RDS?
UPDATE:
I tried to fix the problem by Service:
apiVersion: v1
kind: Service
metadata:
name: postgres-service
spec:
type: ExternalName
externalName: rds-vpc.unique_id.us-east-1.rds.amazonaws.com
So I created this service in EKS, and then tried to refer to postgres-service as DB URL instead of direct RDS host address.
This fix does not work :(
Have you tried to enable "dns propagation" in the peering connection? It looks like you are not getting the internally routable dns. You can enable it by going into the setting for the peering connection and checking the box for dns propagation. I generally do this will all of the peering connections that I control.
The answer I provided here may actually apply to your case, too.
It is about using Services without selectors. Look also into ExternalName Services.