I tried to create a managed certificate for my ingress with this yaml:
---
apiVersion: "networking.gke.io/v1beta1"
kind: "ManagedCertificate"
metadata:
name: "example-cert-webapi"
spec:
domains:
- "foobar.domain.com"
It was successfully created but when I try to describe the said managed certificate using this command:
kubectl describe managedcertificate example-cert-api
It does not show the status. I was expecting that it will be in provisioning status but it the output of the describe command does not show the status. Below is the describe output:
Name: example-cert-webapi
Namespace: default
Labels: <none>
Annotations: <none>
API Version: networking.gke.io/v1
Kind: ManagedCertificate
Metadata:
Creation Timestamp: 2020-09-27T08:31:12Z
Generation: 1
Resource Version: foobarResourceVersion
Self Link: fooBarSelfLink
UID: fooBarUID
Spec:
Domains:
foobar.domain.com
Events: <none>
I have replaced with foobar the entries which are think are sensitive data.
I have also a Cloud DNS setup which corresponds to the domains which I used in the certificate.
Has anyone experienced the same situation? When my ingress deployment finished, the SSL certificate did not take effect.
Thanks in advance!
We noticed the same issue since yesterday. I can confirm that downgrading to 1.16 solved the problem.
Edit: the issue is created at Google: https://issuetracker.google.com/issues/169595857
Related
When I delete a service and recreate, I've noticed that status of the ingress indicates Some backend services are in UNKNOWN state.
After some trials and errors, it seems to be related to name of network endpoint group(NEG). NEG tied with a new service has different name, but the ingress gets an old NEG as backend services.
Then, I found that they works again after I recreate an Ingress.
I'd like to avoid downtime to recreate an ingress as much as possible.
Is there a way to avoid recreating ingress when recreating services?
My Service
apiVersion: v1
kind: Service
metadata:
name: client-service
labels:
app: client
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 80
protocol: TCP
selector:
app: client
My Ingress
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: static-ip-name
networking.gke.io/managed-certificates: managed-certificate
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: client-service
servicePort: 80
If you want to re-use the ingress when the service disappears, you can edit its configuration instead of deleting and recreating it.
To reconfigure the Ingress you will have to update it by editing the configuration, as specified in the official Kubernetes documentation.
To do this, you can perform the following steps:
Issue the command kubectl edit ingress test
Perform the necessary changes, like updating the service configuration
Save the changes
kubectl will update the resource, and trigger an update on the load balancer.
Verify the changes by executing the command kubectl describe ingress test
I am trying to get letsencrypt work with GKE LB, I know there are GCP Managed Certs but it will not work with internal LB as the challenge will not get passed. Letsencrypt DNS certification using cert-manager is there and ready to be used.
❯ k get secrets letsencrypt-prod -o yaml
apiVersion: v1
data:
tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBdlVZTVhXdGNZZUJpMkdadzljRFRLNzY==
kind: Secret
metadata:
creationTimestamp: "2021-01-24T15:03:39Z"
name: letsencrypt-prod
namespace: elastic-system
resourceVersion: "3636289"
selfLink: /api/v1/namespaces/elastic-system/secrets/letsencrypt-prod
uid: f4bec5a9-d3b5-4f4a-9ec6-01a4ce3ba47c
type: Opaque
spec:
tls:
- hosts:
- staging.example.com
- staging2.example.com
secretName: letsencrypt-prod
GCP Reporting this error Error syncing to GCP: error running load balancer syncing routine: error getting secrets for Ingress: secret "letsencrypt-prod" does not specify cert as string data
can anybody help me with what it is missing?
As per this, you must provide a valid format for GCP, like this from your already provided Let's Encrypt valid certs:
kubectl create secret generic letsencrypt-prod --from-file=tls.crt="cert.pem" --from-file=tls.key="privkey.pem" --dry-run -o yaml > output
kubectl apply -f output
Also, (it seems you are already using it, but better safe than sorry), you must define this in the tls section of your Ingress as per this
Actually, it is missed in doc or I am missing as example uses the same name everywhere as metadata.
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: cert-example
namespace: example
spec:
secretName: REAL_NAME_OF_SECRET << This need to include in ingress.
issuerRef:
name: letsencrypt-prod
dnsNames:
- 'staging.domain.com'
- '*.staging.domain.com'
so REAL_NAME_OF_SECRET you should put in ingress or anywhere, where you want to use tls.crt or tls.key.
My setup: Running EKS 1.14 with Calico and Istio 1.5.
Trying to get the sample bookinfo to run with specific NetworkPolicies.
I've applied a GlobalNetworkPolicy that denies all traffic:
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: default-deny
spec:
selector: all()
types:
- Ingress
- Egress
I also added a GlobalNetworkPolicy for Istio to namespace and intra-namespace traffic:
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: allow-istio-system-to-ns
spec:
selector: all()
namespaceSelector: istio-injection == 'enabled'
types:
- Ingress
- Egress
ingress:
- action: Allow
source:
selector: all()
namespaceSelector: projectcalico.org/namespace == 'istio-system'
- action: Allow
source:
selector: all()
namespaceSelector: istio-injection == 'enabled'
egress:
- action: Allow
and a NetworkPolicy allowing all ingress and egress on istio-system
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
name: allow-istio-system-all
namespace: istio-system
spec:
selector: all()
types:
- Ingress
- Egress
ingress:
- action: Allow
egress:
- action: Allow
And a NetworkPolicy to allow traffic to the ingress-gateway on ports 80 and 443. I know this one is redundant, but I was hoping to cut down the previous one to only necessary ingress.
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
name: allow-istio-ingress-on-80-443
namespace: istio-system
spec:
selector: app == 'istio-ingressgateway'
ingress:
- action: Allow
protocol: TCP
destination:
ports:
- 80
- 443
Also some other, unrelated NetworkPolicies (access to kube-dns, metrics-server).
Deploying bookinfo works with and without the policies and same with deploying the Gateway.
The connection between the components works (I can exec into one pod and connect to the others).
But when I try to access the productpage via the gateway with the policies I get nothing back, instead of the HTML I get without the policies.
Am I missing some traffic that should be allowed? Should I have policies for the master nodes or for the loadbalancer somewhere else?
Edit: If I allow all ingress into istio-system and into the namespace, it works. So I think I'm just missing some other ingress from the namespace, is there a way to limit it to just the loadbalancer?
First of all there is a typo in Your allow-istio-system-to-ns yaml:
namespaceSelector: projectcalico.org/namespace == 'istio-system
There should be another ' at the end of the line.
Secondly this could be caused by the changes to policy and mixer in istio version 1.5.
According to Istio documentation:
The mixer policy is deprecated in Istio 1.5 and not recommended for production usage.
Rate limiting: Consider using Envoy native rate limiting instead of mixer rate limiting. Istio will add support for native rate limiting API through the Istio extensions API.
Control headers and routing: Consider using Envoy ext_authz filter, lua filter, or write a filter using the Envoy-wasm sandbox.
Denials and White/Black Listing: Please use the Authorization Policy for enforcing access control to a workload.
There is a guide in istio documentation which allows to turn back on depreciated features:
For an existing Istio mesh
Check the status of policy enforcement for your mesh.
$ kubectl -n istio-system get cm istio -o jsonpath="{#.data.mesh}" | grep disablePolicyChecks
disablePolicyChecks: true
If policy enforcement is enabled (disablePolicyChecks is false), no further action is needed.
Update the istio configuration to enable policy checks.
Execute the following command from the root Istio directory:
$ istioctl manifest apply --set values.global.disablePolicyChecks=false --set values.pilot.policy.enabled=true
configuration "istio" replaced
Validate that policy enforcement is now enabled.
$ kubectl -n istio-system get cm istio -o jsonpath="{#.data.mesh}" | grep disablePolicyChecks
disablePolicyChecks: false
Note that on calico documentation it says it requires the following istio versions:
Istio v1.0, v1.1, v1.2, or v1.3
Hope it helps.
For some reason it seems to work when I replace projectcalico.org/namespace == 'istio-system' with a label from istio-system (e.g. istio-injection=disabled).
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: allow-istio-system-to-ns
spec:
selector: all()
namespaceSelector: istio-injection == 'enabled'
types:
- Ingress
- Egress
ingress:
- action: Allow
source:
selector: all()
namespaceSelector: istio-injection == 'disabled'
- action: Allow
source:
selector: all()
namespaceSelector: istio-injection == 'enabled'
egress:
- action: Allow
I have a requirement where-in I would like to allow certain CIDR ranges to be able to access my service, rest all should be denied.
I have tried the Istio IP Whitelisting/Blacklisting as mentioned in the official Istio documentation.
For example 10.0.0.2/16 should get allowed and rest should be denied. This doesn't seem to work.
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
name: whitelistip
spec:
compiledAdapter: listchecker
params:
# providerUrl: ordinarily black and white lists are maintained
# externally and fetched asynchronously using the providerUrl.
overrides: ["10.0.0.2/16"] # overrides provide a static list
blacklist: true
entryType: IP_ADDRESSES
---
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
name: sourceip
spec:
So basically the istio 1.5.0 was released few days ago and if we check the istio docs white/black list are deprecated now.
Denials and White/Black Listing (Deprecated)
But there is actually a good news because there is new example for authorization on ingress gateway which should answer your question.
I am not able to get the real client IP hence not able to block/allow using authorization policy or IP based whitelisting.
Based on this new example which I tested myself if you want to see you'r source ip you have to change istio-ingressgateway externalTrafficPolicy from Cluster to Local.
Update the ingress gateway to set externalTrafficPolicy: local to preserve the original client source IP on the ingress gateway using the following command:
kubectl patch svc istio-ingressgateway -n istio-system -p '{"spec":{"externalTrafficPolicy":"Local"}}'
And the allow example
The following example creates the authorization policy, ingress-policy, for the Istio ingress gateway. The following policy sets the action field to ALLOW to allow the IP addresses specified in the ipBlocks to access the ingress gateway. IP addresses not in the list will be denied. The ipBlocks supports both single IP address and CIDR notation.
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ingress-policy
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
action: ALLOW
rules:
- from:
- source:
ipBlocks: ["1.2.3.4", "5.6.7.0/24", "$CLIENT_IP"]
I hope this answer your question. Let me know if you have any more questions.
Other solution in Istio 1.5:
Configure external traffic:
kubectl patch svc istio-ingressgateway -n istio-system -p '{"spec":{"externalTrafficPolicy":"Local"}}'
Or in Helm installation:
--set gateways.istio-ingressgateway.externalTrafficPolicy
And you can use in any namespace like this:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: whitelist
namespace: foo
spec:
action: ALLOW
rules:
- from:
- source:
namespaces: ["istio-system"]
when:
- key: request.headers[X-Envoy-External-Address]
values: ["1.2.3.4/24"] #Ip to allow
selector:
matchLabels:
app: httpbin
I have a dockerized flask application that running on kuberneetes in Google Cloud Platform with Identity-Aware Proxy enabled. I can run a "Hello World" website but when I try to use signed JWT headers then problems occur.
In my browser I am presented with
There was a problem with your request. Error code 9
My app is like this example and I use gunicorn to run the app. It seems that trouble happens in the first line
jwt = request.headers.get('x-goog-iap-jwt-assertion')
but that just makes no sense to me. But I can return a string before that line but not after. Any suggestions?
Details on the current kubernetes cluster
apiVersion: apps/v1
kind: Deployment
metadata:
name: internal-tools-app
spec:
selector:
matchLabels:
app: internal-tools
template:
metadata:
labels:
app: internal-tools
spec:
containers:
- name: internal-web-app
image: <<MY_IMAGE>>
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
name: internal-tools-backend-config
namespace: default
spec:
iap:
enabled: true
oauthclientCredentials:
secretName: internal-tools-oauth
---
apiVersion: v1
kind: Service
metadata:
name: internal-tools-service
annotations:
beta.cloud.google.com/backend-config: '{"default": "internal-tools-backend-config"}'
spec:
type: NodePort
selector:
app: internal-tools
ports:
- name: it-first-port
protocol: TCP
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.global-static-ip-name: internal-tools-ip
ingress.gcp.kubernetes.io/pre-shared-cert: "letsencrypt-internal-tools"
name: internal-tools-ingress
spec:
rules:
- host: <<MY_DOMAIN>>
http:
paths:
- backend:
serviceName: internal-tools-service
servicePort: it-first-port
EDIT
Further investigations show
ImportError: Error loading shared library libssl.so.45: No such file or directory (needed by /usr/local/lib/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so)
when running the following
jwt.decode(
iap_jwt, key,
algorithms=['ES256'],
audience=expected_audience)
I just fixed this error code tonight by deleting and recreating my frontend and google-managed cert objects in GCP console. It seems to happen when I decommissioned and repurposed a cluster and deployed my app on a brand new cluster with same static IP address.
I got this answer from the Google Cloud Team bug tracker:
The Error code 9 occurs when multiple requests for re-authentication occur simultaneously (in particular, often caused by browsers reloading multiple windows/tabs at once). This flow currently requires for a temporary cookie flow to succeed first, and this cookie is unique to that flow. However if one flow starts before the previous one finishes, for example with multiple simultaneous refreshes in the same browser, this will cause the error you saw, and cause users to face that auth page.
You can try below options to overcome the issue
reboot 1 browser
clear cookies
better handling of sessions implementing
session handlers
https://issuetracker.google.com/issues/155005454