I am new to kubernetes and want to setup kubernetes HA setup after successfully completing examples with minikube and single master kubernetes cluster. I am using AWS EC2 instances and AWS application load balancer for this purpose. I dont want to use KOPS or any other tool for installation. I want to get hands on with kubeadm.
I followed below steps
Created self signed certificate ca.crt and ca.key to use for
kubernetes
Installed this certificate as Root CA on my ubuntu
instance
Copied this ca.crt and ca.key to /etc/kubernetes/pki
Created new certificate for aws loadbalancer and signed
it with above ca.crt. With this certificate created Aws application
load balancer
I also created Record Set in AWS Route53 for domain name mapping. I also made sure this domain name
mapping is working. (i.e. master.k8sonaws.com is properly resolving to aws load balancer)
Now I am using kubeadm init
kubeadm init --pod-network-cidr=192.168.0.0/20
--service-cidr=192.168.16.0/20 --node-name=10.0.0.13 --control-plane-endpoint "master.k8sonaws.com:443" --upload-certs --v=8 --apiserver-bind-port=443 --apiserver-cert-extra-sans=master.k8sonaws.com,i-0836dd4dc6609a924
This command is succeeding up-to upload-config phase. Health check endpoint is returning success but
after that its failing in upload-config phase
configmaps is forbidden: User "system:anonymous" cannot create resource "configmaps" in API group "" in the namespace
"kube-system"
Here I am not able to understand why kubeadm is passing anonymyous user for api call. How can I resolve this issue
The certificate in the kubeconfig file used to create the ConfigMap does not have right groups. I would say don't generate ca and cert for kubernetes yourself. Just use kubeadm init and kubeadm will handle the ca and cert generation. After your kubernetes cluster is up and running you can use the same ca to generate a cert yourself and use that in aws load balancer.
I found solution to this problem after trying lot of different things
for 2 days. Problem is aws load balancer does not pass client
certificate to backend server when using https listener. And problem is Aws has
not documented this fact or I did not get those document if they are
there.
Way to solve this problem is using plane http listener using same port 443. So SSL termination is carried out by backened server. In my case this is not security threat since both my load balancer and backend servers are internal and not exposed to public ip.
Related
We need to update/renew the API server endpoint (https://****************.__7.region-name.eks.amazonaws.com) for our EKS Cluster due to some security reason and not able to see any option to do that. We have created EKS Cluster back in 2021 and observed that SSL certification validity is of 2 years i.e. around 755 days. so, we wanted to update/renew this certificate.
We tried to validate all certificate (csr file) from kubeadm, but csr file looks fine here. tried to create a cluster configuration file with certificate but seems it was not working with EKS. Please let me knwo if anyone can help us out to figure out how we can update the kuber-apiserver certificate for API server endpoint access of our EKS Cluster.
We are using ECS EC2 as orchestration of docker conatiners.
Also we are using AWS CLOUDMAP/Service discovery to create endpoints of services.
In one of my cluster we are not able to reach endpoint of any service, including the service running in same cluster.It give me below error
closing conenction 0
curl (6) could not resolve host xxx-xxx.test.xxx.org.uk
When i try with the IP instead of domain name like 1x.1XX.7*.2X:port/healtcheck/path it works for all services.
I have check all security groups and NACLS all looks fine.
I setup my ec2 instance and got https working for a bit only to realize I need tls 1.2 on default and in order to do that I had to configure my code to instruct it to read my cert file in the code. Problem is I don't know which it is as there are 269 files in the directory /etc/ssl/certs. I have googled for a couple hours hoping something would tell me where to look to check what file amazon generated for me that it specifically wants. Otherwise im shooting in the dark trying pems one at a time.
secureProtocol: 'TLSv1_2_server_method',
pfx: fs.readFileSync("/etc/ssl/certs/FILENAME.PEM")
}, app).listen(443);
Help is greatly appreciated.
Please refer EC2 instance details in AWS management console.
Steps:
Login to AWS management console and goto EC2 -> Instances.
Select the instance to which we need to connect and scroll the
description which is present in bottom window which will have EC2
instance details.
Check for "Key pair name" , this will be the key pair which needs
to be used to securely connect to respective EC2 instance.
I assume that you got a certificate on Amazon ACM.
ACM Certificates can be used in,
Elastic Load Balancing
Amazon CloudFront
Amazon API Gateway
AWS Elastic Beanstalk
AWS CloudFormation(for email validation only)
The certificate issued by ACM cannot be installed directly on an EC2 instance.
If you want to install an SSL certificate directly on your EC2 instance, you will need to obtain a SSL certificate through a third-party
Therefore, you cannot find any files related to the certificate issued by ACM inside your EC2 instance.
hope this helps.
I am new to Kubernetes. I am using Kops to deploy my Kubernetes application on AWS. I have already registered my domain on AWS and also created a hosted zone and attached it to my default VPC.
Creating my Kubernetes cluster through kops succeeds. However, when I try to validate my cluster using kops validate cluster, it fails with the following error:
unable to resolve Kubernetes cluster API URL dns: lookup api.ucla.dt-api-k8s.com on 149.142.35.46:53: no such host
I have tried debugging this error but failed. Can you please help me out? I am very frustrated now.
From what you describe, you created a Private Hosted Zone in Route 53. The validation is probably failing because Kops is trying to access the cluster API from your machine, which is outside the VPC, but private hosted zones only respond to requests coming from within the VPC. Specifically, the hostname api.ucla.dt-api-k8s.com is where the Kubernetes API lives, and is the means by which you can communicate and issue commands to the cluster from your computer. Private Hosted Zones wouldn't allow you to access this API from the outside world (your computer).
A way to resolve this is to make your hosted zone public. Kops will automatically create a VPC for you (unless configured otherwise), but you can still access the API from your computer.
I encountered this last night using a kops-based cluster creation script that had worked previously. I thought maybe switching regions would help, but it didn't. This morning it is working again. This feels like an intermittency on the AWS side.
So the answer I'm suggesting is:
When this happens, you may need to give it a few hours to resolve itself. In my case, I rebuilt the cluster from scratch after waiting overnight. I don't know whether or not it was necessary to start from scratch -- I hope not.
This is all I had to run:
kops export kubecfg (cluster name) --admin
This imports the "new" kubeconfig needed to access the kops cluster.
I came across this problem with an ubuntu box. What I did was to add the dns record in the hosted zone in route 53 to /etc/hosts.
Here is how I resolved the issue :
Looks like there is a bug with kops library though it shows
**Validation failed: unexpected error during validation: unable to resolve Kubernetes cluster API URL dns: lookup api **
when u try kops validate cluster post waiting for 10-15 mins. Behind the scene the kubernetes cluster is up ! You can verify same by doing ssh in to master node of your kunernetes cluster as below
Go to page where u can ec2 instance and your k8's instances running
copy "Public IPv4 address" of your master k8 node
post login to ec2 instance on command prompt login to master node as below
ssh ubuntu#<<"Public IPv4 address" of your master k8 node>>
Verify if you can see all node of k8 cluster with below command it should show your master node and worker node listed there
kubectl get nodes --all-namespaces
I am creating a k8s cluster on AWS using http://kubernetes.io/docs/getting-started-guides/aws/
After exporting required environment variables and running ./kubernetes/cluster/kube-up.sh I am able to create k8s cluster.
But I am getting "Certificate Error" in crome on accessing kubernetes-master, can some one help me how to specify trusted certificate which is already available with me to kubernetes-master?
During creation I can see logs as,
Generating certs for alternate-names:
Each cluster is given its own certificate hierarchy, which is not signed by a root CA that is trusted by your web browser. For Chrome to connect without a warning, you will need to add the cluster CA certificate into your root certificate store so that it is trusted by your machine.
See https://github.com/kubernetes/kubernetes/issues/7307#issuecomment-96130676