I'm attempting to configure Istio authentication policy to validate our JWT.
I set the policy and can see it takes affect. However it won't allow anything to connect. When applying the policy if I inspect the istio-pilot logs I can see it failing to retrieve the signing keys, giving a certificate error.
2018-10-24T03:22:41.052354Z error model Failed to fetch pubkey from "https://iam.company.com.au/oauth2/jwks": Get https://iam.company.com.au/oauth2/jwks: x509: certificate signed by unknown authority
2018-10-24T03:22:41.052371Z warn Failed to fetch jwt public key from "https://iam.company.com.au/oauth2/jwks "
This I assume would be due to this server using a TLS certificate signed by our corporate CA.
How do I get istio-pilot to trust certs from our CA? I have tried installing ca-certificates and including our CA public key in the Ubuntu certificates but it still won't work.
Policy:
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "our-service-jwt-example"
spec:
targets:
- name: our-service
origins:
- jwt:
issuer: iam.company.com.au
audiences:
- YRhT8xWtcLrOQmqJUGPA1p6O6mUa
jwksUri: "https://iam.company.com.au/oauth2/jwks"
principalBinding: USE_ORIGIN
Pilot does the jwks resolving for the envoy. In that case, pilot needs to have the CA certificate. At the moment there is no way to add a CA cert to the pilot unless you add the cert when deploying pilot in the istio. https://github.com/istio/istio/blob/master/pilot/pkg/model/jwks_resolver.go
This has been added as of Istio 1.4:
https://github.com//istio/istio/pull/17176
You can provide an extra root certificate in PEM format in the pilot.jwksResolverExtraRootCA helm chart value (also works with IstioOperator for more recent versions of Istio) and it will create a ConfigMap containing an extra.pem file that should get mounted into the istio pilot container as /cacerts/extra.pem. From there it should get picked up automatically.
Related
am trying to setup Custom CA (AWS PCA) Integration using Kubernetes CSR in Istio following this doc (Istio / Custom CA Integration using Kubernetes CSR). Steps followed:
i) Enable feature gate on cert-manager controller: --feature-gates=ExperimentalCertificateSigningRequestControllers=true
ii) AWS PCA and aws-privateca-issuer plugin is already in place.
iii) awspcaclusterissuers object in place with AWS PCA arn (arn:aws:acm-pca:us-west-2:<account_id>:certificate-authority/)
iv) Modified Istio operator with defaultConfig and caCertificates of AWS PCA issuer (awspcaclusterissuers.awspca.cert-manager.io/)
v) Modified istiod deployment and added env vars (as mentioned in the doc along with cluster role).
istiod pod is failing with this error:
Generating K8S-signed cert for [istiod.istio-system.svc istiod-remote.istio-system.svc istio-pilot.istio-system.svc] using signer awspcaclusterissuers.awspca.cert-manager.io/cert-manager-aws-root-ca
2023-01-04T07:25:26.942944Z error failed to create discovery service: failed generating key and cert by kubernetes: no certificate returned for the CSR: "csr-workload-lg6kct8nh6r9vx4ld4"
Error: failed to create discovery service: failed generating key and cert by kubernetes: no certificate returned for the CSR: "csr-workload-lg6kct8nh6r9vx4ld4"
K8s Version: 1.22
Istio Version: 1.13.5
Note: Our integration of cert manager and AWS PCA works fine as we generate Private Certificates using cert-manager and PCA with ‘Certificates’ object. It’s the integration of kubernetes csr method with istio that is failing!
Would really appreciate if anybody with knowledge on this could help me out here as there are nearly zero docs on this integration.
I haven't done this with Kubernetes CSR, but I have done it with Istio CSR. Here are the steps to accomplish it with this approach.
Create a certificate in AWS Private CA and download the public root certificate either via the console or AWS CLI (aws acm-pca get-certificate-authority-certificate --certificate-authority-arn <certificate-authority-arn> --region af-south-1 --output text > ca.pem).
Create a secret to store this root certificate. Cert manager will use this public root cert to communicate with the root CA (AWS PCA).
Install cert-manager. Cert manager will essentially function as the intermediate CA in place of istiod.
Install AWS PCA Issuer plugin.
Make sure you have the necessary permissions in place for the workload to communicate with AWS Private CA. The recommended approach would be to use OIDC with IRSA. The other approach is to grant permissions to the node role. The problem with this is that any pod running on your nodes essentially has access to AWS Private CA, which isn't a least privilege approach.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "awspcaissuer",
"Action": [
"acm-pca:DescribeCertificateAuthority",
"acm-pca:GetCertificate",
"acm-pca:IssueCertificate"
],
"Effect": "Allow",
"Resource": "arn:aws:acm-pca:<region>:<account_id>:certificate-authority/<resource_id>"
}
]
}
Once the permissions are in place, you can create a cluster issuer or an issuer that will represent the root CA in the cluster.
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
name: aws-pca-root-ca
spec:
arn: <aws-pca-arn-goes-here>
region: <region-where-ca-was-created-in-aws>
Create the istio-system namespace
Install Istio CSR and update the Helm values for the issuer so that cert-manager knows to communicate with the AWS PCA issuer.
helm install -n cert-manager cert-manager-istio-csr jetstack/cert-manager-istio-csr \
--set "app.certmanager.issuer.group=awspca.cert-manager.io" \
--set "app.certmanager.issuer.kind=AWSPCAClusterIssuer" \
--set "app.certmanager.issuer.name=aws-pca-root-ca" \
--set "app.certmanager.preserveCertificateRequests=true" \
--set "app.server.maxCertificateDuration=48h" \
--set "app.tls.certificateDuration=24h" \
--set "app.tls.istiodCertificateDuration=24h" \
--set "app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.pem" \
--set "volumeMounts[0].name=root-ca" \
--set "volumeMounts[0].mountPath=/var/run/secrets/istio-csr" \
--set "volumes[0].name=root-ca" \
--set "volumes[0].secret.secretName=istio-root-ca"
I would also recommend setting preserveCertificateRequests to true at least for the first time you set this up so that you can actually see the CSRs and whether or not the certificate were successfully issued.
When you install Istio CSR, this will create a certificate called istiod as well as a corresponding secret that stores the cert. The secret is called istiod-tls. This is the cert for the intermediate CA (Cert manager with Istio CSR).
9) Install Istio with the following custom configurations:
Update the CA address to Istio CSR (the new intermediate CA)
Disable istiod from functioning as the CA
Mount istiod with with the cert-manager certificate details
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio
namespace: istio-system
spec:
profile: "demo"
hub: gcr.io/istio-release
values:
global:
# Change certificate provider to cert-manager istio agent for istio agent
caAddress: cert-manager-istio-csr.cert-manager.svc:443
components:
pilot:
k8s:
env:
# Disable istiod CA Sever functionality
- name: ENABLE_CA_SERVER
value: "false"
overlays:
- apiVersion: apps/v1
kind: Deployment
name: istiod
patches:
# Mount istiod serving and webhook certificate from Secret mount
- path: spec.template.spec.containers.[name:discovery].args[7]
value: "--tlsCertFile=/etc/cert-manager/tls/tls.crt"
- path: spec.template.spec.containers.[name:discovery].args[8]
value: "--tlsKeyFile=/etc/cert-manager/tls/tls.key"
- path: spec.template.spec.containers.[name:discovery].args[9]
value: "--caCertFile=/etc/cert-manager/ca/root-cert.pem"
- path: spec.template.spec.containers.[name:discovery].volumeMounts[6]
value:
name: cert-manager
mountPath: "/etc/cert-manager/tls"
readOnly: true
- path: spec.template.spec.containers.[name:discovery].volumeMounts[7]
value:
name: ca-root-cert
mountPath: "/etc/cert-manager/ca"
readOnly: true
- path: spec.template.spec.volumes[6]
value:
name: cert-manager
secret:
secretName: istiod-tls
- path: spec.template.spec.volumes[7]
value:
name: ca-root-cert
configMap:
defaultMode: 420
name: istio-ca-root-cert
If you want to watch a detailed walk-through on how the different components communicate, you can watch this video:
https://youtu.be/jWOfRR4DK8k
In the video, I also show the CSRs and the certs being successfully issued, as well as test that mTLS is working as expected.
The video is long, but you can skip to 17:08 to verify that the solution works.
Here's a repo with these same steps, the relevant manfiest files and architecture diagrams describing the flow: https://github.com/LukeMwila/how-to-setup-external-ca-in-istio
I'm trying to set up GKE Gateway with an HTTPS listener using a wildcard certificate managed via Certificate Manager.
The problem I'm facing is not in provisioning of the certificate, which was done successfully following the DNS Authorization tutorial and this answer. I've successfully provisioned a wildcard certificate, which is shown by gcloud certificate-manager certificates describe <cert-name> as ACTIVE and
AUTHORIZED on both the domain and its wildcard subdomain. I've also provisioned the associated Certificate Map and Map Entry (all via Terraform) and created a global static IP address and a wildcard A record for it in Cloud DNS.
However, when I try to use this cert and address in the GKE Gateway resource, the resource gets "stuck" (never reaches SYNC phase), and there's no HTTPS GCLB provisioned as seen via gcloud or Cloud Console.
Here's the config I was trying to use for it:
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: external-https
annotations:
networking.gke.io/certmap: gke-gateway
spec:
gatewayClassName: gke-l7-gxlb
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
kinds:
- kind: HTTPRoute
- name: https
protocol: HTTPS
port: 443
allowedRoutes:
kinds:
- kind: HTTPRoute
addresses:
- type: NamedAddress
value: gke-gateway
I've tried multiple different combinations of this config, including with an explicit IPAddress, or without allowedRoutes. But no matter what I try, it doesn't seem to work. I can only see the initial ADD and UPDATE events in the output of kubectl describe gateway external-http, and there're no logs for it to be found anywhere afaik (since GKE Gateway Controller is part of the GKE Control Plane and without any logging exposed to the customers, from what I understand).
The only time I was able to make either internal or external Gateway to work is when using HTTP protocol, i.e. without certificates. Hence, I think this has to do with HTTPS, and probably more specifically with linking to the managed wildcard certificate.
Additionally, I should mention that my attempts at deploying the Gateway fail most of the time (i.e. the resource gets "stuck" in the same way), even when reusing a previously-working HTTP config. I'm not sure what the source of this flakiness is (apart from maybe some internal quota), but I imagine this is fully expected, as the service is still in Beta.
Has anyone been able to actually provision a Gateway with HTTPS listener and wildcard certs, and how?
I am new to istio and had doubt configuring a Request authentication policy.The policy uses a jwksuri which is an external URI.The policy is applied on the istio-system namespace.The moment I apply this policy and do
>istioctl proxy-status
The ingress gateway on which the policy is applied LDS is marked stale.If I remove this policy the gateway goes back into SYNCED state.It seems this jwksuri is not accessible since we are behind a company proxy. I created Service entry to access the external jwks uri something like this
kubectl apply -f - <<EOF
apiVersion:
networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: jwksexternal
spec:
hosts:
-
authorization.company.com
ports:
- number: 443
name: https
protocol: HTTPS
resolution: DNS
location: MESH_EXTERNAL
EOF
Also tried to create one more service entry "Configuring traffic to external proxy" referring to this documentation https://istio.io/latest/docs/tasks/traffic-management/egress/http-proxy/
But this is not working.How should I configure the company proxy in Istio.
Edit this is the logs in istiod (Please note https://authorization.company.com/jwk is an external url)
2021-06-02T14:35:39.423938Z error model Failed to fetch public key from "https://authorization.company.com/jwk": Get "https://authorization.company.com/jwk": dial tcp: lookup authorization.company.com on 10.X.0.X:53: no such host
2021-06-02T14:35:39.423987Z error Failed to fetch jwt public key from "https://authorization.company.com/jwk": Get "https://authorization.company.com/jwk": dial tcp: lookup authorization.company.com on 10.X.0.X:53: no such host
2021-06-02T14:35:39.424917Z info ads LDS: PUSH for node:istio-ingressgateway-5b69b5448c-8wbt4.istio-system resources:1 size:4.5kB
2021-06-02T14:35:39.433976Z warn ads ADS:LDS: ACK ERROR router~10.X.48.X~istio-ingressgateway-5b69b5448c-8wbt4.istio-system~istio-system.svc.cluster.local-105 Internal:Error adding/updating listener(s) 0.0.0.0_8443: Provider 'origins-0' in jwt_authn config has invalid local jwks: Jwks RSA [n] or [e] field is missing or has a parse error
Not able to find a workaround for this issue. As of now embedded the jwks into the jwt rules.But this has a problem ,whenever the public key keys get rotated .The jwt rules fail. This is a proxy issue but not sure how to bypass
By default, Istio allows traffic to external systems.
See https://istio.io/latest/docs/tasks/traffic-management/egress/egress-control/#change-to-the-blocking-by-default-policy
So if the problem is that the JWKS URL can't be accessed, it is most likely not because of Istio and a ServiceEntry won't help. I guess the problem will be somewhere else, not in Istio.
I'm trying to use the AWS S3 SDK for Java to connect to a bucket from a Kubernetes pod running an Spring Boot application. In order to get external access I had to create a service as follows:
kind: Service
apiVersion: v1
metadata:
name: s3
namespace: production
spec:
type: ExternalName
externalName: nyc3.digitaloceanspaces.com
And then I modified my configuration in application.properties specifying the endpoint:
cloud.aws.endpoint=s3
cloud.aws.credentials.accessKey=ASD
cloud.aws.credentials.secretKey=123
cloud.aws.credentials.instanceProfile=true
cloud.aws.credentials.useDefaultAwsCredentialsChain=true
Because the SDK builds the host name for the bucket as bucket.s3... I modified my client to use "path style" access with this configuration:
#Bean(name = "amazonS3")
public AmazonS3Client amazonS3Client(AWSCredentialsProvider credentialsProvider,
RegionProvider regionProvider) {
EndpointConfiguration endpointConfiguration = new EndpointConfiguration(
endpoint, regionProvider.getRegion().getName());
return (AmazonS3Client) AmazonS3ClientBuilder.standard()
.withCredentials(credentialsProvider)
.withEndpointConfiguration(endpointConfiguration)
.withPathStyleAccessEnabled(true)
.build();
}
But when I try to perform any bucket operation I get the following error regarding the name mismatch with the SSL certificate:
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <s3> doesn't match any of the subject alternative names: [*.nyc3.digitaloceanspaces.com, nyc3.digitaloceanspaces.com]
How can I avoid this certificate error?
I am having a similar issue. I believe AmazonS3Client API doesn't resolve the k8s service name. I had to directly use a host name instead of K8s service name.
I am trying to setup TLS in kubernetes(DigitalOcean), using cert-manager.
Using Let's Encrypt and certbot on a server machine is well described, but when running in Kubernetes I can not find any information.
I found this but how can I use the certificate in cert-manager on Kubernetes cert-manager ClusterIssuer:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: your_email_address_here
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# Enable the HTTP-01 challenge provider
http01: {}
I dont have any email registret?
you can use any temp mail service but at expiring date of certificate you will not get noticed.
you can use any of you email gmail or anything if you want to setup certificate in Kubernetes using ingress and cert-manager you can follow this link by Digital ocean: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes