I am trying to setup TLS in kubernetes(DigitalOcean), using cert-manager.
Using Let's Encrypt and certbot on a server machine is well described, but when running in Kubernetes I can not find any information.
I found this but how can I use the certificate in cert-manager on Kubernetes cert-manager ClusterIssuer:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: your_email_address_here
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# Enable the HTTP-01 challenge provider
http01: {}
I dont have any email registret?
you can use any temp mail service but at expiring date of certificate you will not get noticed.
you can use any of you email gmail or anything if you want to setup certificate in Kubernetes using ingress and cert-manager you can follow this link by Digital ocean: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes
Related
I have a Kubernetes service with externalIP which points to an RDS instance
The service name is rds
We are using pods with hostname as rds.default.svc.cluster.local to connect to the database using a DNS redirect using the above service. It works fine without SSL but for SSL to work it also needs the Host Name, which in this case is not visible and the certificate check fails.
Is there a way to get the ExternalName from the endpoint rds.default.svc.cluster.local. we know that it works if we directly provide the hostname = but this makes it a lot difficult to manage for each service running in Kubernetes.
Service calling this host is using sequelize to establish a TLS connection
YAML for ExternalIP service
apiVersion: v1
kind: Service
metadata:
name: rds
namespace: default
spec:
externalName: <endpoint>.rds.amazonaws.com
ports:
- name: rds-aurora
port: 5432
protocol: TCP
targetPort: 5432
type: ExternalName
Error
[queryReportMessage]: Error occured: SequelizeConnectionError: Hostname/IP does not match certificate's altnames: Host: rds.default.svc.cluster.local. is not in the cert's altnames: DNS:*.<URL>.rds.amazonaws.com
I created a TLS-enabled service with AWS PCA and cert-manager by this post:
https://aws.amazon.com/blogs/security/tls-enabled-kubernetes-clusters-with-acm-private-ca-and-amazon-eks-2/
After I deployed a demo application with ingress, I tested access on control node
$ curl https://demo.my-org.com --cacert cacert.pem
Got message
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
The cacert.pem was downloaded from AWS PCA's Certificate body. Things look good in K8s for AWSPCAClusterIssuer and Certificate. The certificate description got these events:
$ kubectl describe certificate rsa-cert-2048 -n acm-pca-lab-demo
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 47m cert-manager Existing issued Secret is not up to date for spec: [spec.commonName spec.dnsNames]
Normal Reused 47m cert-manager Reusing private key stored in existing Secret resource "rsa-example-cert-2048"
Normal Requested 47m cert-manager Created new CertificateRequest resource "rsa-cert-2048-pp4c4"
Normal Issuing 47m cert-manager The certificate has been successfully issued
If I access from browser got 502 error. The certificate page shown a fake certificate and an alt DNS name.
I'm sure the private CA in AWS was actived successfully. Its arn and region were been set to the EKS node policy and AWSPCAClusterIssuer. What's wrong about the settings? How to diagnose the issue?
deployed resources
I checked the deployed resources in acm-pca-lab-demo namespace.
$ kubectl get secret -n acm-pca-lab-demo
NAME TYPE DATA AGE
default-token-jmxt7 kubernetes.io/service-account-token 3 10h
rsa-example-cert-2048 kubernetes.io/tls 3 10h
$ kubectl get all -n acm-pca-lab-demo
NAME READY STATUS RESTARTS AGE
pod/hello-world-57df4c69f9-nnjrl 1/1 Running 0 10h
pod/hello-world-57df4c69f9-r8f4p 1/1 Running 0 10h
pod/hello-world-57df4c69f9-xgm6w 1/1 Running 0 10h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hello-world ClusterIP 102.30.45.163 <none> 80/TCP 10h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-world 3/3 3 3 10h
NAME DESIRED CURRENT READY AGE
replicaset.apps/hello-world-57df4c69f9 3 3 3 10h
$ kubectl get ingress -n acm-pca-lab-demo
NAME CLASS HOSTS ADDRESS PORTS AGE
acm-pca-demo-ingress <none> demo.my-org.com 11111111111111111111111111111111-2222222222222222.elb.us-east-1.amazonaws.com 80, 443 10h
On the browser, I also got these messages:
The certificate is not trusted because it is self-signed.
HTTP Strict Transport Security: false
HTTP Public Key Pinning: false
certificate file
I downloaded the PCA .pem file from AWS console here. Is it correct?
It's -----BEGIN CERTIFICATE----- started file.
Check your ingress configuration, share the YAML config-if possible which you have used with application deployment.
there could be chances there is not secret attached to ingress, due to that K8s Nginx ingress controller by default attaching the default FAKE cert instead of your generated cert.
For example :
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: acm-pca-demo-ingress
namespace: acm-pca-lab-demo
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- www.rsa-2048.example.com
secretName: rsa-example-cert-2048
rules:
- host: www.rsa-2048.example.com
http:
paths:
- path: /
pathType: Exact
backend:
service:
name: hello-world
port:
number: 80
as shown above rsa-example-cert-2048, make sure your secret exists in the namespace in which ingress there.
i am testing with Open SSL in GCP instance. and how can generate Self Managed Certificates in GCP instance.
You can make certificate and domain status active, it can take up to 30 mins for your load balancer to begin using your self-managed SSL certificate
To test this you can run the following OpenSSL command, replacing
DOMAIN ----------------------- with-----------------------DNS name
IP_ADDRESS-------------------with-----------------------IP address of your load balancer.
echo | openssl s_client -showcerts -servername DOMAIN -connect IP_ADDRESS:443 -verify 99 -verify_return_error
This command outputs the certificates that the load balancer presents to the client. Along with other detailed information, the output should include the certificate chain.
Verify return code: 0 (ok).
For more information you can refer to this link.
There are so many ways we can issue certificates, let's focus on K8S cluster running on Google (GKE) using a custom resource called ManagedCertificate and ingress rules.
You must own the domain name and name must be no longer than 63 characters.
Create a reserved (static) external IP address using the following
command, or use google console.
gcloud compute addresses create gke-static-ip --global
Create Managed Certificate
---
apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
name: gke-certificate
spec:
domains:
- DOMAIN
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gke-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: gke-static-ip
networking.gke.io/managed-certificates: gke-certificate
spec:
backend:
serviceName: hello-world-service
servicePort: 80
in my case i use the cloud endpoints as domain name
Project Prod and Project Staging have been setup and each running a GKE cluster. Cert-manager is installed to automate the process of certificate issuance as explained in official docs.
Project Prod has DNS that maps to both prod and staging cluster istio gateway IP addresses.
On DNS01 challenge for cluster in Project Prod, manages to authenticate, and certificate is issued successfully.
But the cluster running in Project Staging, fails to get certificate due to not enough permission to authenticate and verify via Cloud DNS setup in Project Prod.
In Project Prod, there is a service account with dns/admin role that is setup via GKE secret and accessed in clusterissuer like so
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod-clusterissuer
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: name#name.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod-clusterissuer
solvers:
# ACME DNS-01 provider configurations
- dns01:
# Google Cloud DNS
cloudDNS:
# Secret from the google service account key
serviceAccountSecretRef:
name: cloud-dns-key
key: key.json
# The project in which to update the DNS zone
project: iprocure-server-prod
Certificate is issued successfully in Project Prod GKE cluster.
Project Staging GKE cluster, clusterissuer has its service account with dns/admin role just like in Project Prod, but fails to perform dns01 challenge in Project Prod DNS.
Following error is seen when kubectl describe challenge
Type Reason Age From Message
---- ------ ---- ---- -------
Warning PresentError 2m56s (x19 over 7h14m) cert-manager Error presenting challenge: GoogleCloud API call failed: googleapi: Error 403: Forbidden, forbidden
What should be done to Project Staging service account to enable dns01 challenge to be performed in Project Prod Clous DNS
I faced this problem too. I had to replicate the service account in my Prod Project with DNS ADMIN permission to Staging Project so that the GKE cluster in Staging can have enough permission to authenticate and verify via Cloud DNS setup in the Prod project
You have to create a SA in Prod Project with DNS ADMIN permission and master the email of that SA then go to project B and make that SA email a member by adding it as a member with also DNS ADMIN permission.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: {{ .Values.app.certificate.issuer.name }}
namespace: {{ .Values.app.namespace }}
labels:
app.kubernetes.io/managed-by: "Helm"
spec:
acme:
email: {{ .Values.app.certificate.acme.email }}
privateKeySecretRef:
name: {{ .Values.app.certificate.issuer.name }}
server: {{ .Values.app.certificate.acme.server }}
solvers:
- dns01:
cloudDNS:
project: {{ .Values.app.project_id }} ## Make sure this is the project of where DNS is e.g Prod Project
serviceAccountSecretRef:
name: {{ .Values.secrets.name }}
# Secret from the google service account key
key: key.json
I have a domain (example.com) already configured in Cloud DNS. With this domain I can access microservices that are in a GKE cluster. I use istio-ingressgateway IP in CloudDNS to make the association between the cluster
Now I have another domain (newexample.com) with a custom certificate for https connections. Is there a way to redirect all the requests to newexample.com to example.com? I do not want to change anything in gke/istio configuration if possible.
Each method will require some of the reconfiguration in either GKE/Istio side.
One of the solutions to this is to have a CNAME record in a Cloud DNS and a SSL certificate with Alternative Names.
With above solution you will be able to send requests to your GKE/Istio cluster with both domain names assuming correct Istio configuration.
What is CNAME?
CNAME is a Canonical Name Record or Alias Record.
A type of resource record in the Domain Name System (DNS), that specifies that one domain name is an alias of another canonical domain name.
Example of a CNAME record:
DNS name Type TTL Data
old.domain. A 60 1.2.3.4
new.domain. CNAME 60 old.domain.
Alternative Names:
A SAN or subject alternative name is a structured way to indicate all of the domain names and IP addresses that are secured by the certificate.
Enstrustdatacard.com: What is a san and how is it used
You can create SSL cerificate create to support both:
old.domain
new.domain
There are plenty options to do that for example Let's Encrypt or Cert Manager.
Example
I've created an example to show you how to do it:
Configure DNS zone in Cloud DNS
Create a basic app with a service
Create a certificate for example app
Create Istio resources to allow connections to example app
Test
Configure DNS zone in Cloud DNS
You will need to have 2 records:
A record with IP of your Ingress Gateway and name: old.domain
CNAME record pointing to old.domain with name: new.domain
Please take a look on official documentation: Cloud.google.com: DNS: Records
Create a basic app with a service
Below is example app with a service which will respond with a basic hello:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-dp
spec:
selector:
matchLabels:
app: hello-dp
replicas: 1
template:
metadata:
labels:
app: hello-dp
spec:
containers:
- name: hello
image: "gcr.io/google-samples/hello-app:2.0"
env:
- name: "PORT"
value: "50001"
---
apiVersion: v1
kind: Service
metadata:
name: hello-sv
spec:
selector:
app: hello-dp
ports:
- name: hello-port
protocol: TCP
port: 50001
targetPort: 50001
type: ClusterIP
Create a certificate for example app
As said previously, certificate with Alternative Names can be created with Let's Encrypt. I created it with:
GCE VM with Ubuntu 16.04
Open port 80
Domain name old.domain pointing to public ip address of a VM
Guide: Linode.com: Docs: Install let's encrypt to create a SSL certificate
Command to create certificate:
$ ./letsencrypt-auto certonly --standalone -dold.domain-dnew.domain
Certificate created in /etc/letsencrypt/archive/ used in creating tls secret for GKE with command:
$ kubectl create secret tlsssl-certificate--cert cert1.pem --key privkey1.pem
Please have in mind that this certificate was created only for testing purposes and I strongly advise using dedicated solution like: Cert-manager
PS: If you used this method please revert back changes in the Cloud DNS to point the Istio gateway.
Create Istio resources to allow connections to example app
Below are example Istio resources allowing connections to example app with support for HTTPS:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: hello-gw
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: ssl-certificate
hosts:
- "old.domain"
- "new.domain"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: hello-vs
spec:
hosts:
- "old.domain"
- "new.domain"
gateways:
- hello-gw
http:
- route:
- destination:
host: hello-sv
port:
number: 50001
Please take a specific look on:
tls:
mode: SIMPLE
credentialName: ssl-certificate
This part will ensure that connection to the cluster will use HTTPS
Additionally:
hosts:
- "old.domain"
- "new.domain"
Above definition in both resources will allow only connections with specified domains.
Test
When applied all of the above resources you should be able to enter in your browser:
https://old.domain
https://new.domain
and be greeted with below message and valid SSL certificate:
Hello, world!
Version: 2.0.0
Hostname: hello-dp-5dd8b85b56-bk7zr