I'm trying to setup a Service Entry to add an external API to our mesh and take advantage of some network resilience features. First, I'd like to add retries to these API calls. I've got an example setup and working using http, but I can't figure out the configuration for https.
For example, I got this working...
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: httpbin-srv-entry
spec:
hosts:
- httpbin.org
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_EXTERNAL
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin-vrt-srv
spec:
hosts:
- httpbin.org
http:
- route:
- destination:
host: httpbin.org
retries:
attempts: 10
retryOn: "5xx"
After making a call inside the mesh to curl -v http://httpbin.org/status/503, I see all of the attempts in the istio-proxy log:
tools istio-proxy 2022-05-06T20:52:39.327431Z debug envoy router [C940][S13424897362262611751] performing retry
tools istio-proxy 2022-05-06T20:52:39.381075Z debug envoy router [C940][S13424897362262611751] performing retry
tools istio-proxy 2022-05-06T20:52:39.467414Z debug envoy router [C940][S13424897362262611751] performing retry
tools istio-proxy 2022-05-06T20:52:39.647938Z debug envoy router [C940][S13424897362262611751] performing retry
tools istio-proxy 2022-05-06T20:52:39.681137Z debug envoy router [C940][S13424897362262611751] performing retry
tools istio-proxy 2022-05-06T20:52:39.713335Z debug envoy router [C940][S13424897362262611751] performing retry
tools istio-proxy 2022-05-06T20:52:39.831212Z debug envoy router [C940][S13424897362262611751] performing retry
tools istio-proxy 2022-05-06T20:52:40.006095Z debug envoy router [C940][S13424897362262611751] performing retry
tools istio-proxy 2022-05-06T20:52:40.189391Z debug envoy router [C940][S13424897362262611751] performing retry
tools istio-proxy 2022-05-06T20:52:40.287682Z debug envoy router [C940][S13424897362262611751] performing retry
The problem I'm having is setting up this example with https...
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: httpbin-srv-entry
spec:
hosts:
- httpbin.org
ports:
- number: 443
name: https
protocol: HTTPS
location: MESH_EXTERNAL
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin-vrt-srv
spec:
hosts:
- httpbin.org
http:
- route:
- destination:
host: httpbin.org
retries:
attempts: 10
retryOn: "5xx"
The API calls are working, but the retires aren't happening. I can tell by how quick the curl command ends and there's nothing in the istio-proxy log indicating any retries happened. I also tested by spinning up kennethreitz/httpbin on a different cluster and watching the access log, just thought this example was easier to demo.
Figured this out from here: https://stackoverflow.com/a/51740657/510218
Full example:
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: httpbin-srv-entry
spec:
hosts:
- httpbin.org
ports:
- number: 80
name: http
protocol: HTTP
- number: 443
name: https
protocol: HTTPS
location: MESH_EXTERNAL
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin-vrt-srv
spec:
hosts:
- httpbin.org
http:
- match:
- port: 80
route:
- destination:
host: httpbin.org
port:
number: 443
timeout: 300s
retries:
attempts: 10
retryOn: 5xx
perTryTimeout: 3s
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: originate-tls-for-httpbin
spec:
host: httpbin.org
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: SIMPLE # initiates HTTPS when accessing httpbin.org
Related
I have backend and fronted applications. I have tried to create one ingress for fronted where both paths will be matched (host.com/api/v1/reference/list/test1 and host.com/api/v1/reference/test2). The second one works fine, but the first give me error: Failed to load resource: the server responded with a status of 500 (). Here is my ingress configuration:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: backend-app
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2$3
spec:
tls:
- hosts:
- host.com
secretName: tls-secret
rules:
- host: host.com
http:
paths:
- backend:
serviceName: service-backend
servicePort: 80
path: /api(/|$)(.*)
service:
apiVersion: v1
kind: Service
metadata:
name: service-backend
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
Does anyone know why my URLs are not getting rewritten and the requests are not delivered to the backend service for host.com/api/v1/reference/test2 ?
Thanks in advance!
resolved - bug inside application
ISTIO version: 1.9.4
EKS Cluster version: 1.14
We have deployed ISTIO APP mesh in our project. We have deployed External Authorization using istio's documentation i.e. https://istio.io/latest/docs/tasks/security/authorization/authz-custom/.
External authorizer used (as mentioned in above documentation) : https://raw.githubusercontent.com/istio/istio/release-1.9/samples/extauthz/ext-authz.yaml
When we access any API from going into pod of another API (i.e. over http), using curl command, all works fine. External auth service gets call and all the headers are passed into external authorizer's v3 check method. Below information is passed
source, principal, destination, headers: authority, method, path, accept, content-length, user-agent, x-b3-sampled, x-b3-spanid, x-b3-traceid, x-envoy-attempt-count, x-ext-authz, x-forwarded-client-certx-forwarded-proto, x-request-id.
But when we try to access the same service over https using postman, browser or from going into pod of another API and using curl with https endpoint, we get denied response from external authorizer's v3 check method. Also when we check the logs of external authorizer's v3 check method no headers are passed to it in this case.
Below is setup
Name spaces with ISTIO ejection enable : foo
1. ISTIO Config map changes
data:
mesh: |-
# Add the following content to define the external authorizers.
extensionProviders:
- name: "sample-ext-authz-grpc"
envoyExtAuthzGrpc:
service: "ext-authz.foo.svc.cluster.local"
port: "9000"
- name: "sample-ext-authz-http"
envoyExtAuthzHttp:
service: "ext-authz.foo.svc.cluster.local"
port: "8000"
includeHeadersInCheck: ["x-ext-authz"]
2. External Authorizer
apiVersion: v1
kind: Service
metadata:
name: ext-authz
namespace: foo
labels:
app: ext-authz
spec:
ports:
- name: http
port: 8000
targetPort: 8000
- name: grpc
port: 9000
targetPort: 9000
selector:
app: ext-authz
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ext-authz
namespace: foo
spec:
replicas: 1
selector:
matchLabels:
app: ext-authz
template:
metadata:
labels:
app: ext-authz
spec:
containers:
- image: docker.io/istio/ext-authz:0.6
imagePullPolicy: IfNotPresent
name: ext-authz
ports:
- containerPort: 8000
- containerPort: 9000
3. Enable the external authorization Config
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ext-authz
namespace: foo
spec:
selector:
matchLabels:
app: user-api
action: CUSTOM
provider:
name: sample-ext-authz-grpc
rules:
- to:
- operation:
paths: ["/user/api/*"]
4. PeerAuth Chagnes
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: mtlsauth
namespace: foo
spec:
mtls:
mode: STRICT
5. Destination Rule
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: default
namespace: foo
spec:
host: "*.samplehost.svc.cluster.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
6. Virtual Service File
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: sample-gateway
namespace: foo
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "sample.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: user-api
namespace: foo
spec:
hosts:
- "sample.com"
gateways:
- sample-gateway
http:
- match:
- uri:
prefix: /user/api/
route:
- destination:
host: user-api
port:
number: 9500
Logs from ingress gateway:
2021-07-08T11:13:33.554104Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp 172.20.0.51:15012: connect: connection refused"
2021-07-08T11:13:35.420052Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-07-08T11:43:24.012961Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0
I am not sure if you are facing the issue but if seems like you have enforced mtls . Thats why in the following config for gateway. You might need to open HTTPS also
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: sample-gateway
namespace: foo
spec:
selector:
istio: ingressgateway
servers:
port:
number: 80
name: http
protocol: HTTP
hosts:
"sample.com"
port:
number: 443
name: https
protocol: HTTPS
hosts:
"sample.com"
Following this doc I got istio-ingressgateway running but using curl to test the URL I am facing this problem:
curl: (7) Failed to connect to httpbin.example.com port 31390: Connection refused
This is the Gateway:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: mygateway
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: httpbin-credential # must be the same as secret
hosts:
- httpbin.example.com
and the virtual service:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- "httpbin.example.com"
gateways:
- mygateway
http:
- match:
- uri:
prefix: /status
- uri:
prefix: /delay
route:
- destination:
port:
number: 8000
host: httpbin
I am using istio 1.5.4.
This is the command that result in timeout:
curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"
Goal
I'm trying to setup a
Cloud LB -> GKE [istio-gateway -> my-service]
This was working before, however, I have to recreate the cluster 2 days ago and run into this problem. Maybe some version change?
This is my ingress manifest file
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: "my-dev-ingress"
namespace: "istio-system"
annotations:
kubernetes.io/ingress.global-static-ip-name: "my-dev-gclb-ip"
ingress.gcp.kubernetes.io/pre-shared-cert: "my-dev-cluster-cert-05"
kubernetes.io/ingress.allow-http: "false"
spec:
backend:
serviceName: "istio-ingressgateway"
servicePort: 80
Problem
The health check issue by the Cloud LB failed. The backend service created by the Ingress create a /:80 default health check.
What I have tried
1) I tried to set the health check generated by the gke ingress to point to the istio-gateway StatusPort port 15020 in the Backend config console. Then the health check passed for a bit until the backend config revert itself to use the original /:80 healthcheck that it created. I even tried to delete the healthcheck that it created and it just create another one.
2) I also tried using the istio-virtual service to route the healthcheck to 15020 port as shown here with out much success.
3) I also tried just route everything in the virtual-service the healthcheck port
hosts:
- "*"
gateways:
- my-web-gateway
http:
- match:
- method:
exact: GET
uri:
exact: /
route:
- destination:
host: istio-ingress.gke-system.svc.cluster.local
port:
number: 15020
4) Most of the search result I found say that setting readinessProbe in the deployment should tell the ingress to set the proper health check. However, all of my service are under the istio-gateway and I can't really do the same.
I'm very lost right now and will really appreciate it if anyone could point me to the right direction. Thanks
i got it working with gke 1.20.4-gke.2200 and istio 1.9.2, the documentation for this is non existent or i have not found anything, you have to add an annotation to istio-ingressgateway service to use a backend-config when using "istioctl install -f values.yaml" command
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
components:
ingressGateways:
- name: istio-ingressgateway
enabled: true
k8s:
serviceAnnotations:
cloud.google.com/backend-config: '{"default": "istio-ingressgateway-config"}'
then you have to create the backend-config with the correct healthcheck port
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: istio-ingressgateway-config
namespace: istio-system
spec:
healthCheck:
checkIntervalSec: 30
port: 15021
type: HTTP
requestPath: /healthz/ready
with this the ingress should automatically change the configuration for the load balancer health check pointing to istio port 80
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web
namespace: istio-system
annotations:
kubernetes.io/ingress.global-static-ip-name: web
networking.gke.io/managed-certificates: "web"
spec:
rules:
- host: test.example.com
http:
paths:
- path: "/*"
pathType: Prefix
backend:
service:
name: istio-ingressgateway
port:
number: 80
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: direct-web
namespace: istio-system
spec:
hosts:
- test.example.com
gateways:
- web
http:
- match:
- uri:
prefix: "/"
route:
- destination:
port:
number: 8080 #internal service port
host: "internal-service.service-namespace.svc.cluster.local"
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: web
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- test.example.com
you could also set hosts to "*" in the virtualservice and gateway
I am trying to expose kiali on my default gateway. I have other services working for apps in the default namespace but have not been able to route traffic to anything in the istio namespace
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- '*'
tls:
httpsRedirect: true
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- '*'
tls:
mode: SIMPLE
privateKey: /etc/istio/ingressgateway-certs/tls.key
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: kiali
namespace: default
spec:
hosts:
- kiali.dev.example.com
gateways:
- gateway
http:
- route:
- destination:
host: kiali.istio-system.svc.cluster.local
port:
number: 20001
The problem was I had mTLS enabled and kiali does not have a sidecar thus can not be validated by mTLS. the solution was to add a destination rule disabling mTLS for it.
apiVersion: 'networking.istio.io/v1alpha3'
kind: DestinationRule
metadata:
name: kiali
namespace: istio-system
spec:
host: kiali.istio-system.svc.cluster.local
trafficPolicy:
tls:
mode: DISABLE
You should define an ingress gateway and make sure that the hosts in the gateway match the hosts in the virtual service. Also specify the port of the destination. See the Control Ingress Traffic task.
For me this worked!
I ran
istioctl proxy-config routes istio-ingressgateway-866d7949c6-68tt4 -n istio-system -o json > ./routes.json
to get the dump of all the routes. The kiali route got corrupted for some reason. I deleted the virtual service and created it again, that fixed it.
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: kiali
namespace: istio-system
spec:
gateways:
- istio-system/my-gateway
hosts:
- 'mydomain.com'
http:
- match:
- uri:
prefix: /kiali/
route:
- destination:
host: kiali.istio-system.svc.cluster.local
port:
number: 20001
weight: 100
---
apiVersion: 'networking.istio.io/v1alpha3'
kind: DestinationRule
metadata:
name: kiali
namespace: istio-system
spec:
host: kiali.istio-system.svc.cluster.local
trafficPolicy:
tls:
mode: SIMPLE
---
Note: hosts needed to be set, '*' didnt work for some reason.