I have a quick question on Kubernetes ingress. I have both Nginx ingress controller and AWS ALB ingress controller and ingress resources for both Nginx and AWS ALB in a single cluster. Both of these ingress resources are pointed to a single service and deployment file, meaning, bothe the ingress resources are pointed to the same service. However, when I hit the Nginx ingress URL, I'm able to see the desired page, but with the AWS ALB ingress, I can only see the apache default page. I know this doesn't sound practical, but I'm trying to test out something with both these ingress resources. Just wanted to understand, where am I missing out on seeing the application for AWS ALB ingress URL.
Posting this community wiki answer to point that the issue to this question was resolved in the comments.
Feel free to edit and expand.
The solution to the issue:
AWS ALB Ingress was pointing to the default apache document root in the pod. I modified the document root to the application data and was able to see my application page open up!
Additional resources that could be useful in this particular example:
Tecmint.com: Change root directory of a apache web server - how to change the apache2: DocumentRoot
Github.com: Kubernetes sigs: AWS load balancer controller: Docs: Examples - examples of ALB that satisfies Ingress resource.
Related
I deployed Istio in a Kubernetes cluster hosted in AWS EKS. That created a Kubernetes service of type LoadBalancer named istio-ingressgateway with an external hostname of [redacted]-redacted.us-west-2.elb.amazonaws.com, and automatically provisioned an AWS ELB, type classic. That's great. (Note: in AWS console I do not see a hosted zone for this elb hostname, it doesn't look like I can configure aliases or whatnot)
I've been able to connect a gRPC client running outside the cluster, to a gRPC server running in the cluster by defining an istio gateway which opens port 80 to host:"*", and defining an istio virtual service which routes port 80 to my destination (some port of some kubernetes service). So far so good.
Now I would like to do this for a second gRPC endpoint in the cluster. As far as I know, my choices are: either route by opening a second port (say 81) on the ingress (which I'm choosing not to do for now). Or route by defining subdomains of [redacted]-redacted.us-west-2.elb.amazonaws.com, or route by implementing "virtual hosting", i.e. two services on the same domain where the client sends host:"svc1" in a header.
This is where I'm stuck. I don't know how do define subdomains to this ELB domain name, and I don't know if I can do virtual hosting with TLS, and if so, can I still use the "passthrough" tls mode of Istio so as not to TLS-terminate on the gateway.
1- Using subdomains: When on my developer machine I try getent hosts svc1.[redacted]-redacted.us-west-2.elb.amazonaws.com it does not map to an IP, whereas if remove the svc1 prefix DNS returns 3 IPs (guessing the 3 availability zones in us-west-2). So I can't prefix svc1 to the domain without some work.
Can I define subdomains svc1 and svc2 for a domain created by classic AWS ELB automatically provisioned by deploying Istio, and if so how? Can I do this with kubectl/istioctl to configure Istio, or do I need to do this with the AWS CLI? Can this be done without registering a domain, i'm fine with that user-unfriendly elb.amazonaws.com hostname?
2- Virtual Hosting: Alternatively I could add a host=svc1 in the gRPC metadata. I tried adding ctx := metadata.AppendToOutgoingContext(context.Background(), "Host", "svc1") to my Go client, that did not work.
Some guidance would be great.
Update 1, and one solution for plaintext
I'm reading on List of HTTP header fields the "Host" header should not be used in HTTP2. Given this is gRPC it is HTTP2 and as such I should not add this as a custom header in my gRPC client requests. The solution when doing virtual hosting, in a gRPC client instead of sending a "host" header, you send an ":Authority" header, the value must correspond to the host specified in istio's gateway+virtualservice. This works for me. In my Go client I had to add dialOptions = append(dialOptions, grpc.WithAuthority("[my-service-1]")).
That solution won't work for TLS according to GoDoc grpc.WithAuthority. So I still need to find how to route 2 services from the same external IP and port when TLS is used.
Update 1.1:
In virtual hosting, when TLS is used, the SNI can be passed in the CLIENT HELLO message, instead of being a gRPC header like in plaintext . Depending on your language the exact API is going to vary, but in Go I was able to pass the host name by setting the ServerName field in tls.Config. You can then gate on that hostname in your istio gateway configuration, and route based on that value in your istio virtualservice.
Update 2: I tried, and failed, to create the subdomain on AWS Route53. First I create a public hosted zone for domain:[redacted]-redacted.us-west-2.elb.amazonaws.com. Next, in that hosted zone I create an A record with name=svc1.[redacted]-redacted.us-west-2.elb.amazonaws.com, route=Alias to Classic Load Balancer, region=us-west-2, load balancer=dualstack.[redacted]-redacted.us-west-2.elb.amazonaws.com.
I later test the svc1.[redacted]-redacted.us-west-2.elb.amazonaws.com A record exists with dig, which does not return an answer. However dig does return 3 A records for [redacted]-redacted.us-west-2.elb.amazonaws.com (without the svc1 prefix),with the IPs of 3 load balancers.
Istio
Can I define subdomains svc1 and svc2 for a domain created by classic AWS ELB automatically provisioned by deploying Istio, and if so how?
You can't do it with istio, you have to configure that in the cloud, in your case you have to configure that on aws.
On istio you can only specify the hosts, which would be the subdomains configured on aws.
Virtual Services would look like this:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: mipnw-vs1
spec:
hosts:
- svc1.example.com
http:
- name: "svc1-route"
match:
- uri:
prefix: /
route:
- destination:
host: svc1.default.svc.cluster.local
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: mipnw-vs2
spec:
hosts:
- svc2.example.com
http:
- name: "svc2-route"
match:
- uri:
prefix: /
route:
- destination:
host: svc2.default.svc.cluster.local
Aws
I would love to learn how to create 2 subdomains "abc" and "def" for domain "[redacted]-[redacted].us-west-2.elb.amazonaws.com"
I'm not exactly sure that's what you're looking for, but I did some research in the aws documentation and I found that you could use Amazon Route 53 to achieve what you need.
Amazon Route 53 is a highly available and scalable cloud Domain Name System (DNS) web service. It is designed to give developers and businesses an extremely reliable and cost effective way to route end users to Internet applications by translating names like www.example.com into the numeric IP addresses like 192.0.2.1
The following informations are collected from:
https://aws.amazon.com/route53/faqs/
https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-elb-load-balancer.html
If you host a website on multiple Amazon EC2 instances, you can distribute traffic to your website across the instances by using an Elastic Load Balancing (ELB) load balancer. The ELB service automatically scales the load balancer as traffic to your website changes over time. The load balancer also can monitor the health of its registered instances and route domain traffic only to healthy instances.
To route domain traffic to an ELB load balancer, use Amazon Route 53 to create an alias record that points to your load balancer. An alias record is a Route 53 extension to DNS. It's similar to a CNAME record, but you can create an alias record both for the root domain, such as example.com, and for subdomains, such as www.example.com. (You can create CNAME records only for subdomains.)
Amazon Route 53 also offers alias records, which are an Amazon Route 53-specific extension to DNS. You can create alias records to route traffic to selected AWS resources, including Amazon Elastic Load Balancing load balancers, Amazon CloudFront distributions, AWS Elastic Beanstalk environments, API Gateways, VPC interface endpoints, and Amazon S3 buckets that are configured as websites. Alias record typically have a type of A or AAAA, but they work like a CNAME record. Using an alias record, you can map your record name (example.com) to the DNS name for an AWS resource(elb1234.elb.amazonaws.com). Resolvers see the A or AAAA record and the IP address of the AWS resource.
And there is Question about the subdomains
Can I use 'Alias' records with my sub-domains?
Yes. You can also use Alias records to map your sub-domains (www.example.com, pictures.example.com, etc.) to your ELB load balancers, CloudFront distributions, or S3 website buckets.
Additional resources:
https://medium.com/cloud-native-the-gathering/istio-ingress-to-expose-your-k8s-services-via-individual-dns-2ec9c2717b81
https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-routing-traffic-for-subdomains.html
We are using EKS and Nginx-ingress(NLB). I'm trying to configure multiple AWS ACM certificates in the AWS-load-balancer-SSL-cert annotation for NLB. But with no luck. Could someone help if it possible at all? Thanks
If that not possible, Please guide me any other way on how to use multiple ACM cert in the ingress object-level if possible.
My EXACT Scenario:-
I am using an NLB (FYI)
If we able to add multiple ACM certificate at controller level that also works for me (I am using a single certificate in my NLB currently see below annotations)
At the controller level, these flags help me to add a single certificate:-
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:ap-south-1:1234556677:certificate/3a1d5a-469b-dffe4bad3182
service.beta.kubernetes.io/aws-load-balancer-type: nlb
or
I am maintaining an ingress object as per NameSpace. if we are able to attach a Certificate at the ingress object level, which also solves my problem.
Good question.
There is no support for multiple ACM certificates on an ALB/NLB that points to an nginx ingress controller (or any other ingress controller AFAIK).
The dirty hack from Kubernetes is to create another Service that points to the same nginx ingress controller (same selectors) but in this case, it will just create another ALB/NLB and you may not want that.
The non-Kubernetes way which is the way might work better for you is just to do it from AWS itself and modify the ALB/NLB that sends traffic to your nginx ingress.
✌️
To add in Rico's answer.
It's not possible to attach multiple certificates to the Nginx ingress controller or any other ingress with annotation : service.beta.kubernetes.io/aws-load-balancer-ssl-cert.
Closed PR : https://github.com/kubernetes/kubernetes/pull/95208
Issue thread: https://github.com/kubernetes/cloud-provider-aws/issues/80#issuecomment-686722657
It's not working with NLB However if you are using the ALB you can use this annotation
Single cert with ALB
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:xxxxx:certificate/xxxxxxx
Multiple certificates
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:xxxxx:certificate/cert1,arn:aws:acm:us-west-2:xxxxx:certificate/cert2,arn:aws:acm:us-west-2:xxxxx:certificate/cert3
alb.ingress.kubernetes.io/certificate-arn specifies the ARN of one or more certificates managed by ACM
Another Option For NLB
Create an ACM certificate with multiple Wild card domains and use this single Cert with ingress. this will work with NLB also
So your ACM certificate will be storing certs for multiple domains example
*.example.com
*.hello.io
*.so.in
single ACM certificate now you can use with NLB Ingress, and no need worry about attaching multiple certs.
Option : 2 using cert-manager and storing cert in secret
It would be better if you planning to use multiple domains use wild card certificates with Cert-manager store them into Secret of K8s and use it as pluggable solution with ingress.
I think both the other answers are old and not valid now as I got confused reading those as they mention that its not possible to add 2 certs for NLB. However, you can easily add certs on nginx ingress controller with the annotation in a comma separated string. I myself have deployed AWS LB Controller and using nginx ingress controller to deploy NLB. My annotations on nginx ingress controller service
service.beta.kubernetes.io/aws-load-balancer-ssl-cert=arn:aws:acm:eu-central-1:123456:certificate/123abc,arn:aws:acm:eu-central-1:123456:certificate/123xyz
and I have 2 ACM certs attached to my load balancer both with separate URLs
abc.com
xyz.com
I am using kops to deploy my kubernetes cluster. in my cluster, I have a simple Nginx deployment, a service, and an ingress. Its configured with route 53 (not using external DNS, manually creating A record in route 53 pointing to a classic load balancer (generated by kops)).
I can hit the domain www.XXXX.com as well as a subdomain on it but, there is not SSL certificate on it.
I know we can apply SSL on the loadbalancer. So, went in ACM, created a certificate and when I try to apply it, I see 80 and 443 are serving TCP traffic, that's why I cannot add these two ports serving HTTP and HTTPS with certificate (as shown in image)
If I delete the two TCP listeners and add HTTP and HTTPS listners I can, but then my app is not reachable on the domain.
How can I configure ACM on this loadbalancer. Is this even a correct way of adding https for an app deployed on Kubernetes?
I know about https://kubernetes.github.io/ingress-nginx/user-guide/tls/, but if there is a way to do it using above approach I would prefer that.
UPDATE 1:
when I describe my nginx pod I see Ports as follows:
Ports: 80/TCP, 443/TCP
shouldn't that be
Ports: 80/HTTP, 443/HTTPS
?
Turns out I deployed the ingress controller incorrectly.
the documentation clearly says download and update values in the file before applying I missed the updating part.
You have to update CIDR proxy-real-ip-cidr and service.beta.kubernetes.io/aws-load-balancer-ssl-cert
literally that all you need.
make sure you are creating a certificate before applying this YAML file.
and while creating cert add
DOMAIN.com
*.DOMAIN.com
both to make it work.
I got yaml file for specifying ssl certificate (provided by aws certificate manager)to load balancer for kubernetes deployment. But, we are running kubernetes cluster in aws china account where certification manager option is not available. Now if I have SSL certificate provided by Godaddy, how can I install it? Is any other alternative ways to install certificate rather than load balancer? Can I install it in my tomcat container itself and build new image with it?
As far as I know, you cannot setup an ELB deployed with a kubernetes Service to use a certificate which is NOT an ACM certificate. In fact, if you take a look at the possibile annotations here you'll see that the only annotation available to select a certificate is service.beta.kubernetes.io/aws-load-balancer-ssl-cert and the documentation for that annotation says the following:
ServiceAnnotationLoadBalancerCertificate is the annotation used on the
service to request a secure listener. Value is a valid certificate ARN.
For more, see http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-listener-config.html
CertARN is an IAM or CM certificate ARN, e.g. arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012
As you ask, you can for sure terminate your ssl inside your kubernetes Pod and make the ELB a simple TCP proxy.
In order to do so, you need to add the following annotation to your Service manifest:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: 'tcp'
Also, you will need to forward both your http and https ports in order to handle http to https redirect correctly inside you pod.
If you need more specific help, please post you current manifest.
I'm trying to setup IAP with a HTTPS load balancer as per instructions here: https://cloud.google.com/iap/docs/load-balancer-howto
My backend is gke cluster that has a ingress on port 80 to access http web server.
Frontend is https with a valid certificate.
The traffic is routed without any issues from LB to web server through HTTPS FE, but when I want to enable IAP using command as below:
gcloud beta compute backend-services update k8s-be-30324--34c500f0e91c741a --iap=enabled --global
It returns the following output:
WARNING: IAP only protects requests that go through the Cloud Load Balancer. See the IAP documentation for important security best practices: https://cloud.google.com/iap/
WARNING: IAP has been enabled for a backend service that does not use HTTPS. Data sent from the Load Balancer to your VM will not be encrypted.
ERROR: (gcloud.beta.compute.backend-services.update) There was a problem modifying the resource:
- Invalid value for field 'resource.iap': ''. Backend service with IAP enabled requires at least one HTTPS proxy.
Any advice is appreciated! Thanks
So I figured out a workaround is to use the same LB that is created with ingress for kubernetes cluster instead using a custom one. Of course to avoid leaking unauthorized access FE for http must be removed from the LB.