I've had great difficulties routing traffic to k8s API and services.
First I've created a cluster(k8s.buycheese.com) with KOPS in private
topology, within a VPC, so that, master and nodes are only accessible from a bastion using SSH.
I own a domain in namecheap (buycheese.com) and I've created a hosted zone(k8s.buycheese.com) in route53.
After KOPS has installed the cluster, it added a couple of record sets to the hosted zone like api.k8s.buycheese.com.
I've added the hosted zone's namespaces to my domain in namecheap, so that I can access the Kubernetes cluster(kubectl). That works correctly!
Next, I've installed an ingress nginx controller. Then I've created 2 ingresses:
One to expose the Kubernetes dashboard
Another one to expose a nodeJS application
I then tested my nodeJS Application using the ingress nginx ELB's URL and I can confirm that works! So I know that my pods are running correctly and the ELB works fine!
But obviously, I want my applications to be accessed through the domain I own...
So basically:
I need a new subdomain dashboard.buycheese.com to get to the Kubernetes dashboard.
And I need buycheese.com and www.buycheese.com domains to redirect to my nodeJS app.
Well, to do that, I've created a new hosted zone named buycheese.com in route53, and added 4 new namespaces to my domain buycheese.com in namecheap.
Then I've created 2 aliases(A) within that same hosted zone:
dashboard.buycheese.com with Alias Target: ingress nginx's ELB
www.buycheese.com with Alias Target: ingress nginx's ELB
Then within my 2 ingress files
# Dashboard
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
name: kubernetes-dashboard-oidc
namespace: kube-system
spec:
rules:
- host: dashboard.buycheese.com
http:
paths:
- path: /
backend:
serviceName: kubernetes-dashboard-oidc
servicePort: 80
# NodeJS App
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
name: app
namespace:default
spec:
rules:
- host: buycheese.com
http:
paths:
- path: /
backend:
serviceName: app-service
servicePort: 3000
To sum up I have 2 hosted zones
1) k8s.buycheese.com
2) buycheese.com
2 Alias within hosted zone buycheese.com:
1) www.buycheese.com
2) dashboard.buycheese.com
2 Ingresses to expose the dashboard and my app
That configuration does not work at all! The below URLs are not reachable!
dashboard.buycheese.com
www.buycheese.com
buycheese.com
Only the ELB's URL works!
So first I would like to know whether my set up is correct(obviously no, but why ?)
What's the right way to make all of those URLs exposing my services and applications?
Thanks for your help!
The only thing that I think may be happening here is that Alias Target: is not forwarding to the ELB.
You can try using CNAME records instead. You can always test using dig from the command line:
$ dig buycheese.com
$ dig www.buycheese.com
$ dig dashboard.buycheese.com
I created new record set in Route 53 with Type=A and alias to my ELB DNS Name for me also initially the same problem was there but after waiting for some time it started working this could be DNS propagation issue as told by denixtry
Related
I have multiple deployments running of RDP application and they all are exposed with ClusterIP service. I have nginx-ingress controller in my k8s cluster and to allow tcp I have added --tcp-services-configmap flag in nginx-ingress controller deployment and also created a configmap for the same that is shown below
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: ingress-nginx
data:
3389: “demo/rdp-service1:3389”
This will expose “rdp-service1” service. And I have 10 more such services which needed to be exposed on the same port number but if I add more service in the same configmap like this
...
data
3389: “demo/rdp-service1:3389”
3389: “demo/rdp-service2:3389”
Then it will remove the previous service data and since here I have also deployed external-dns in k8s, so all the records created by ingress using host: ... will starts pointing to the deployment attached with the newly added service in configmap.
Now my final requirement is as soon as I append the rule for a newly created deployment(RDP application) in the ingress then it starts allowing the TCP connection for that, so is there any way to achieve this. Or is there any other Ingress controller available that can solve such type of use case and can also easily be integrated with external-dns ?
Note:- I am using AWS EKS Cluster and Route53 with external-dns.
Posting this answer as a community wiki to explain some of the topics in the question as well as hopefully point to the solution.
Feel free to expand/edit it.
NGINX Ingress main responsibility is to forward the HTTP/HTTPS traffic. With the addition of the tcp-services/udp-services it can also forward the TCP/UDP traffic to their respective endpoints:
Kubernetes.github.io: Ingress nginx: User guide: Exposing tcp udp services
The main issue is that the Host based routing for Ingress resource in Kubernetes is targeting specifically HTTP/HTTPS traffic and not TCP (RDP).
You could achieve a following scenario:
Ingress controller:
3389 - RDP Deployment #1
3390 - RDP Deployment #2
3391 - RDP Deployment #3
Where there would be no Host based routing. It would be more like port-forwarding.
A side note!
This setup would also depend on the ability of the LoadBalancer to allocate ports (which could be limited due to cloud provider specification)
As for possible solution which could be not so straight-forward I would take a look on following resources:
Stackoverflow.com: Questions: Nxing TCP forwarding based on hostname
Doc.traefik.io: Traefik: Routing: Routers: Configuring TCP routers
Github.com: Bolkedebruin: Rdpgw
I'd also check following links:
Aws.amazon.con: Quickstart: Architecture: Rd gateway - AWS specific
Docs.konghq.com: Kubernetes ingress controller: 1.2.X: Guides: Using tcpingress
Haproxy:
Haproxy.com: Documentation: Aloha: 12-0: Deployment guides: Remote desktop: RDP gateway
Haproxy.com: Documentation: Aloha: 10-5: Deployment guides: Remote desktop
Haproxy.com: Blog: Microsoft remote desktop services rds load balancing and protection
Actually, I really don't know why you are using that configmap.
In my knowledge, nginx-ingress-controller is routing traffic coming in the same port and routing based on host. So if you want to expose your applications on the same port, try using this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: {{ .Chart.Name }}-ingress
namespace: your-namespace
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: your-hostname
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: {{ .Chart.Name }}-service
servicePort: {{ .Values.service.nodeport.port }}
Looking in your requirement, I feel that you need a LoadBalancer rather than Ingress
I am trying to set up a service and expose it externally on EKS. I have already done it on GKE pretty easily but now AWS is giving me a hard time.
My NGINX yaml looks something like that:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myapp-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- app.mydomain.com
secretName: myapp-tls
rules:
- host: app.mydomain.com
http:
paths:
- path: /
backend:
serviceName: myapp-service
servicePort: 80
And then I have my domain app.mydomain.com on Google Domains pointing at the ingress external address. There is also a cert-manager service running in order to support HTTPS.
However, while basically the same setup worked completely out of the box on GKE, EKS gives me a hard time.
From what I understand it has something to do with EKS default LoadBalancer being layer 4 in comparison to Google's layer 7 (Which explains HTTPS not working) but there is also issues with redirections of the domain as it just resolves as the ingress address instead of my desired address and thus my app doesn't show up.
The domain is registered over Google Domains and I'm creating Synthetic Records (for my subdomain) that points to my ingress external address on EKS. The same scheme works perfectly fine on GKE but here it resolves the address as the ingress address instead of my domain which results in 404 on the ingress side.
I was wondering if someone could please point me to how to properly set it up? Should I give up on nginx ingress on EKS and move onto ALB? and how to properly associate the domain?
Thank you very much in advance!
Edit:
output of kubectl describe ingress myapp-ingress:
Name: myapp-ingress
Namespace: default
Address: ********************************-****************.elb.eu-west-1.amazonaws.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
myapp-tls terminates app.mydomain.com
Rules:
Host Path Backends
---- ---- --------
app.mydomain.com
/ myapp-service:80 (172.31.2.238:8000)
Annotations: cert-manager.io/cluster-issuer: myapp-letsencrypt-prod
kubernetes.io/ingress.class: nginx
Events: <none>
Should I give up on nginx ingress on EKS and move onto ALB
No. NGinX ingress controllers work perfectly well on EKS. It is possible to configure them as either layer 4 or layer 7; we use it in layer 7 mode.
Can you update your question with the output of
kubectl get ingress myapp-ingress
I think your ingress path is also incorrect. Unless I'm mistaken that's just routing the root of your app, not all uris. We use the scheme
spec:
rules:
- host: service.d.tld
http:
paths:
- path: /?(.*) # <---
backend:
serviceName: my-service
servicePort: http
Are you seeing errors in the nginx ingress controller's logs? That + kubectl events are both useful for debugging purposes.
I'd disable TLS everywhere and get your service working on http, then work stepwise on getting TLS enabled on the ingress controller.
Edit: Based on your response above,
curl -H "Host: app.mydomain.com" http://<elb-address>:80
SHOULD call through to your service behind the ingress.
How is app.mydomain.com defined? Is it a CNAME to the dns entry?
I followed this tutorial https://cloud.google.com/storage/docs/hosting-static-website
But I am not able to reach the site on https because of ERR_SSL_VERSION_OR_CIPHER_MISMATCH / SSL_ERROR_NO_CYPHER_OVERLAP depending on the browser
I use managed certificate provided by google, but no browser seems to be compatible with it. I use GCP default SSL policy, but I also tried create one for testing with minimal requirements of TSL 1.0, but nothing changed.
Yes , if using google managed cert sometimes it takes time to propagate to your associate domain , so in future you could eiter use "curl" command or used dig command to verify it , sometimes it takes 24 hrs too which is maximum time.
Please verify following points:
verify your website pointing towards frontend LB
check the state of google managed cert added on the front end of LB
verify that frontend is using HTTPS and backend is using HTTP
verify your ssl cert
I'm sure it's a problem with the DNS server. If your config is correct, you have to wait a few hours more and redeploy again.
In my case, I was setting up a subdomain with different IP used in the domain.
My managed certificate it was something like this:
apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
name: my-certificate
spec:
domains:
- www.sub.example.com
My ingress was fine:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: st-ext-ip-prod
networking.gke.io/v1beta1.FrontendConfig: ssl-redirect
networking.gke.io/managed-certificates: my-certificate
spec: ...
The problem was the configuration in my DNS SERVER. In my certificate, I was using the domain starting with wwww but in my server, I didn't have the CNAME to support the www.sub
# A DOMAIN-IP
www CNAME example.com
sub A SUBDOMAIN-IP
www.sub CNAME sub.example.com
Doing that configuration (CNAME for www.sub) I had to wait like 5 hours (it could take more)
I had to redeploy everything from the beginning and finally, I didn't have that issue ERR_SSL_VERSION_OR_CIPHER_MISMATCH again.
I have a domain (example.com) already configured in Cloud DNS. With this domain I can access microservices that are in a GKE cluster. I use istio-ingressgateway IP in CloudDNS to make the association between the cluster
Now I have another domain (newexample.com) with a custom certificate for https connections. Is there a way to redirect all the requests to newexample.com to example.com? I do not want to change anything in gke/istio configuration if possible.
Each method will require some of the reconfiguration in either GKE/Istio side.
One of the solutions to this is to have a CNAME record in a Cloud DNS and a SSL certificate with Alternative Names.
With above solution you will be able to send requests to your GKE/Istio cluster with both domain names assuming correct Istio configuration.
What is CNAME?
CNAME is a Canonical Name Record or Alias Record.
A type of resource record in the Domain Name System (DNS), that specifies that one domain name is an alias of another canonical domain name.
Example of a CNAME record:
DNS name Type TTL Data
old.domain. A 60 1.2.3.4
new.domain. CNAME 60 old.domain.
Alternative Names:
A SAN or subject alternative name is a structured way to indicate all of the domain names and IP addresses that are secured by the certificate.
Enstrustdatacard.com: What is a san and how is it used
You can create SSL cerificate create to support both:
old.domain
new.domain
There are plenty options to do that for example Let's Encrypt or Cert Manager.
Example
I've created an example to show you how to do it:
Configure DNS zone in Cloud DNS
Create a basic app with a service
Create a certificate for example app
Create Istio resources to allow connections to example app
Test
Configure DNS zone in Cloud DNS
You will need to have 2 records:
A record with IP of your Ingress Gateway and name: old.domain
CNAME record pointing to old.domain with name: new.domain
Please take a look on official documentation: Cloud.google.com: DNS: Records
Create a basic app with a service
Below is example app with a service which will respond with a basic hello:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-dp
spec:
selector:
matchLabels:
app: hello-dp
replicas: 1
template:
metadata:
labels:
app: hello-dp
spec:
containers:
- name: hello
image: "gcr.io/google-samples/hello-app:2.0"
env:
- name: "PORT"
value: "50001"
---
apiVersion: v1
kind: Service
metadata:
name: hello-sv
spec:
selector:
app: hello-dp
ports:
- name: hello-port
protocol: TCP
port: 50001
targetPort: 50001
type: ClusterIP
Create a certificate for example app
As said previously, certificate with Alternative Names can be created with Let's Encrypt. I created it with:
GCE VM with Ubuntu 16.04
Open port 80
Domain name old.domain pointing to public ip address of a VM
Guide: Linode.com: Docs: Install let's encrypt to create a SSL certificate
Command to create certificate:
$ ./letsencrypt-auto certonly --standalone -dold.domain-dnew.domain
Certificate created in /etc/letsencrypt/archive/ used in creating tls secret for GKE with command:
$ kubectl create secret tlsssl-certificate--cert cert1.pem --key privkey1.pem
Please have in mind that this certificate was created only for testing purposes and I strongly advise using dedicated solution like: Cert-manager
PS: If you used this method please revert back changes in the Cloud DNS to point the Istio gateway.
Create Istio resources to allow connections to example app
Below are example Istio resources allowing connections to example app with support for HTTPS:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: hello-gw
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: ssl-certificate
hosts:
- "old.domain"
- "new.domain"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: hello-vs
spec:
hosts:
- "old.domain"
- "new.domain"
gateways:
- hello-gw
http:
- route:
- destination:
host: hello-sv
port:
number: 50001
Please take a specific look on:
tls:
mode: SIMPLE
credentialName: ssl-certificate
This part will ensure that connection to the cluster will use HTTPS
Additionally:
hosts:
- "old.domain"
- "new.domain"
Above definition in both resources will allow only connections with specified domains.
Test
When applied all of the above resources you should be able to enter in your browser:
https://old.domain
https://new.domain
and be greeted with below message and valid SSL certificate:
Hello, world!
Version: 2.0.0
Hostname: hello-dp-5dd8b85b56-bk7zr
I have a couple of questions
When we make changes to ingress resource, are there any cases where we have to delete the resource and re-create it again or is kubectl apply -f <file_name> sufficient?
When I add the host attribute without www i.e. (my-domain.in), I am not able to access my application but with www i.e. (www.my-domain.in) it works, what's the difference?
Below is my ingress resource
When I have the host set to my-domain.in, I am unable to access my application, but when i set the host to www.my-domain.in I can access the application.
my domain is on a different provider and I have added CNAME (www) pointing to DNS name of my ALB.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: eks-learning-ingress
namespace: production
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/certificate-arn: arn:aws:a982529496:cerd878ef678df
labels:
app: eks-learning-ingress
spec:
rules:
- host: my-domain.in **does not work**
http:
paths:
- path: /*
backend:
serviceName: eks-learning-service
servicePort: 80
First answering your question 1:
When we make changes to ingress resource, are there any cases where we have to delete the resource and re-create it again or is kubectl apply -f sufficient?
In theory, yes, the kubectl apply is the correct way, either it will show ingress unchanged or ingress configured.
Other valid documented option is kubectl edit ingress INGRESS_NAME which saves and apply at the end of the edition if the output is valid.
I said theory because bugs happen, so we can't fully discard it, but bug is the worst case scenario.
Now the blurrier question 2:
When I add the host attribute without www i.e. (my-domain.in), I am not able to access my application but with www i.e. (www.my-domain.in) it works, what's the difference?
To troubleshoot it we need to isolate the processes, like in a chain we have to find which link is broken. One by one:
Endpoint > Domain Provider> Cloud Provider > Ingress > Service > Pod.
DNS Resolution (Domain Provider)
DNS Resolution (Cloud Provider)
Kubernetes Ingress (Ingress > Service > Pod)
DNS Resolution
Domain Provider:
To the Internet, who answers for my-domain.in is your Domain Provider.
What are the rules for my-domain.in and it's subdomains (like www.my-domain.in or admin.my-domain.in)?
You said "domain is on a different provider and I have added CNAME (www) pointing to DNS name of my ALB."
Are my-domain.in and my-domain.in being redirected to the ALB address instinctively?
How does it handle URL subdomains? how the request is passed on to your Cloud?
Cloud Provider:
Ok, the cloud provider is receiving the request correctly and distinctly.
Does your ALB have generic or specific rules for subdomains or path requests?
Test with another host, a different VM with a web server.
Check ALB Troubleshooting Page
Kubernetes Ingress
Usually we would start the troubleshoot from this part, but since you mentioned it works with www.my-domain.in, we can presume that your service, deployment and even ingress structure is working correctly.
You can check the Types of Ingress Docs to get a few examples of how it should work.
Bottom Line: I believe your DNS has a route for www.my-domain.in but the root domain has no route to your cloud provider that's why it's only working when you are enabling the ingress for www.