Istio traffic routing rules take no effect - istio

I am trying to configure a request routing using Istio and Ingress-nginx but I'm not able to route the requests properly. Basically I have two deployments each as a different subset and implemented a weighted VirtualService.
In Kiali dashboard it shows the request being routed from the ingress-controller to PassthroughCluster even though I can see the correct route mapping using istioctl proxy-config routes command.
Here is the complete configuration:
apiVersion: v1
kind: ServiceAccount
metadata:
name: dummy-app
namespace: my-namespace
---
apiVersion: v1
kind: Service
metadata:
name: dummy-app
namespace: my-namespace
labels:
app: dummy-app
service: dummy-app
spec:
ports:
- port: 8080
targetPort: 8080
name: http-web
selector:
app: dummy-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dummy-app-1
namespace: my-namespace
spec:
replicas: 1
selector:
matchLabels:
app: dummy-app
version: v1
template:
metadata:
annotations:
sidecar.istio.io/inject: "true"
labels:
app: dummy-app
version: v1
spec:
serviceAccountName: dummy-app
containers:
- image: my-img
imagePullPolicy: IfNotPresent
name: dummy-app
env:
- name: X_HTTP_ENV
value: dummy-app-1
ports:
- containerPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dummy-app-2
namespace: my-namespace
spec:
replicas: 1
selector:
matchLabels:
app: dummy-app
version: v2
template:
metadata:
annotations:
sidecar.istio.io/inject: "true"
labels:
app: dummy-app
version: v2
spec:
serviceAccountName: dummy-app
containers:
- image: my-img
imagePullPolicy: IfNotPresent
name: dummy-app
env:
- name: X_HTTP_ENV
value: dummy-app-2
ports:
- containerPort: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: dummy-app
namespace: my-namespace
spec:
host: dummy-app
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: dummy-app
namespace: my-namespace
spec:
hosts:
- dummy-app.my-namespace.svc.cluster.local
http:
- match:
- uri:
prefix: "/my-route"
route:
- destination:
host: dummy-app.my-namespace.svc.cluster.local
subset: v1
weight: 0
- destination:
host: dummy-app.my-namespace.svc.cluster.local
subset: v2
weight: 100
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: "my-ingress-class"
nginx.ingress.kubernetes.io/service-upstream: "true"
nginx.ingress.kubernetes.io/upstream-vhost: dummy-app.my-namespace.svc.cluster.local
name: dummy-ingress
namespace: my-namespace
spec:
rules:
- host: myapp.com
http:
paths:
- backend:
service:
name: dummy-app
port:
number: 8080
path: /my-route(.*)
pathType: ImplementationSpecific
Weird thing is I have other applications working in the same namespace and using the same ingress-controller, so I'm not considering there is a problem with ingress-nginx setup.
Istio version:
client version: 1.11.4
control plane version: 1.11.4
data plane version: 1.11.4 (13 proxies), 1.12-dev (15 proxies)
Any ideas on what is the configuration problem or how can I better debug these kind of issues in Istio?

Main issue seems to be with ingress-nginx resource. Based on the above ingress definition, you are trying to bypass istio ingress gateway (where all the proxying rules has been implemented, like gateway,virtual-service and destination rules) and directly pushing the traffic to the application service from ingress. For istio proxy rules to work, you should let traffic pass through istio-ingressgateway (a service under istio-system namespace). So following changes should be made to your ingress resource:
rules:
- host: myapp.com
http:
paths:
- backend:
service:
name: istio-ingressgateway.istio-system
port:
number: 80
path: /my-route(.*)
pathType: ImplementationSpecific

Related

upstream connect error or disconnect/reset before headers. reset reason: connection failure

I'm facing this issue upstream connect error or disconnect/reset before headers. reset reason: connection failure here the my deployment and service file
apiVersion: v1
kind: Service
metadata:
name: project
labels:
app: project
service: project
spec:
ports:
- port: 9080
name: http
selector:
app: project
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: project-svc
labels:
account: project
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: project-v1
labels:
app: project
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: project
version: v1
template:
metadata:
labels:
app: project
version: v1
spec:
serviceAccountName: project-svc
containers:
- name: project
image: segullshairbutt/website:admin_project_a_01_cl1_wn1_pod1_c4
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
and here are the Gateway and virtualservice
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: project-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: projectinfo
spec:
hosts:
- "*"
gateways:
- project-gateway
http:
- match:
- uri:
exact: /productpage
route:
- destination:
host: project
port:
number: 9080
when i visit using minikube-ip:istio-engress i get this error but when I just change the image from my to bookinfo product-page ther nothing this error. I don't know why this is and from where.
Please help me I'll be very thankful to you!

Istio spec/hosts don't work in non-default namespace

I have the following for stg namespace:
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: stg
labels:
app: nginx
spec:
ports:
- port: 80
name: http
selector:
app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: stg
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:alpine
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: gateway
namespace: stg
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- '*'
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: nginx-1
namespace: stg
spec:
hosts:
- "*"
gateways:
- gateway
http:
- route:
- destination:
host: nginx
port:
number: 80
I need spec/hosts to be app1.my.domain instead of *
It works fine in default namespace but not in stg namespace
stg namespace has istio-injection=enabled
Any ideas why it doesn't work and how to fix it?

Traffic routing has issue on AWS but works on Minikube using below config

Using this config Getting No healthy upstream Http 503. If i just remove subset everything works perfectly fine.
Source: ccgf-helm-umbrella-chart/charts/ccgf-cdlg-app/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: ccgf-cdlg-app
service: ccgf-cdlg-app
name: ccgf-cdlg-app
namespace: cdlg-edc-devci
spec:
selector:
app: ccgf-cdlg-app
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
---
Source: ccgf-helm-umbrella-chart/charts/ccgf-cdlg-app/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ccgf-cdlg-app-production
namespace: cdlg-edc-devci
labels:
app: ccgf-cdlg-app
version: production
spec:
replicas: 1
selector:
matchLabels:
app: ccgf-cdlg-app
version: production
template:
metadata:
labels:
app: ccgf-cdlg-app
version: production
spec:
containers:
- image: edc-ccgf-ui-app:1.37
imagePullPolicy: Always
name: ccgf-cdlg-app
ports:
- name: ccgf-cdlg-app
containerPort: 80
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 20
periodSeconds: 20
imagePullSecrets:
- name: spinnakerrepoaccess
Source: ccgf-helm-umbrella-chart/charts/ccgf-cdlg-app/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ccgf-cdlg-app-canary
namespace: cdlg-edc-devci
labels:
app: ccgf-cdlg-app
version: canary
spec:
replicas: 1
selector:
matchLabels:
app: ccgf-cdlg-app
version: canary
template:
metadata:
labels:
app: ccgf-cdlg-app
version: canary
spec:
containers:
- image: edc-ccgf-ui-app:1.38
imagePullPolicy: Always
name: ccgf-cdlg-app
ports:
- name: ccgf-cdlg-app
containerPort: 80
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 20
periodSeconds: 20
imagePullSecrets:
- name: spinnakerrepoaccess
#virtual Service
kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
name: ccgf-cdlg-app
namespace: cdlg-edc-devci
spec:
hosts:
- '*'
gateways:
- ccgf-gateway
http:
- match:
- uri:
prefix: /cdlg-edc-devci/frontend
rewrite:
uri: /
route:
- destination:
host: ccgf-cdlg-app.cdlg-edc-devci.svc.cluster.local
subset: production
retries:
attempts: 3
perTryTimeout: 2s
retryOn: 'gateway-error,connect-failure,refused-stream'
weight: 50
- destination:
host: ccgf-cdlg-app.cdlg-edc-devci.svc.cluster.local
subset: canary
retries:
attempts: 3
perTryTimeout: 2s
retryOn: 'gateway-error,connect-failure,refused-stream'
weight: 50
- match:
- uri:
prefix: /static
rewrite:
uri: /static
route:
- destination:
host: ccgf-cdlg-app.cdlg-edc-devci.svc.cluster.local
retries:
attempts: 3
perTryTimeout: 2s
retryOn: 'gateway-error,connect-failure,refused-stream'
#Destination rule
kind: DestinationRule
apiVersion: networking.istio.io/v1alpha3
metadata:
name: ccgf-cdlg-app
namespace: cdlg-edc-devci
spec:
host: ccgf-cdlg-app
subsets:
- labels:
version: canary
name: canary
- labels:
version: production
name: production
Source: ccgf-helm-umbrella-chart/charts/ccgf-gateway/templates/gateway.yaml
kind: Gateway
apiVersion: networking.istio.io/v1alpha3
metadata:
name: ccgf-gateway
namespace: namespace
spec:
servers:
- hosts:
- '*'
port:
name: http
number: 80
protocol: HTTP
selector:
release: istio-custom-ingress-gateways
I made a reproduction based on your yamls and everything works just fine, the only thing I have is basic istio ingress gateway instead of the custom one.
For start could you please change Host in DestinationRule and check if it works then?
It should be
ccgf-cdlg-app.cdlg-edc-devci.svc.cluster.local instead of ccgf-cdlg-app
Did you enable istio injection in your cdlg-edc-devci namespace?
You can check it with kubectl get namespace -L istio-injection
It should be
NAME STATUS AGE ISTIO-INJECTION
cdlg-edc-devci Active 37m enabled
And the reproduction yamls.
kubectl create namespace cdlg-edc-devci
kubectl label namespace cdlg-edc-devci istio-injection=enabled
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx1
namespace: cdlg-edc-devci
spec:
selector:
matchLabels:
app: ccgf-cdlg-app
version: production
replicas: 1
template:
metadata:
labels:
app: ccgf-cdlg-app
version: production
spec:
containers:
- name: nginx1
image: nginx
ports:
- name: http-dep1
containerPort: 80
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello nginx1 > /usr/share/nginx/html/index.html"]
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx2
namespace: cdlg-edc-devci
spec:
selector:
matchLabels:
app: ccgf-cdlg-app
version: canary
replicas: 1
template:
metadata:
labels:
app: ccgf-cdlg-app
version: canary
spec:
containers:
- name: nginx2
image: nginx
ports:
- name: http-dep2
containerPort: 80
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello nginx2 > /usr/share/nginx/html/index.html"]
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: cdlg-edc-devci
labels:
app: ccgf-cdlg-app
spec:
ports:
- name: http-svc
port: 80
protocol: TCP
selector:
app: ccgf-cdlg-app
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: nginxvirt
namespace: cdlg-edc-devci
spec:
gateways:
- ccgf-gateway
hosts:
- '*'
http:
- name: production
match:
- uri:
prefix: /cdlg-edc-devci/frontend
rewrite:
uri: /
route:
- destination:
host: nginx.cdlg-edc-devci.svc.cluster.local
port:
number: 80
subset: can
weight: 50
- destination:
host: nginx.cdlg-edc-devci.svc.cluster.local
port:
number: 80
subset: prod
weight: 50
- name: canary
match:
- uri:
prefix: /s
rewrite:
uri: /
route:
- destination:
host: nginx.cdlg-edc-devci.svc.cluster.local
port:
number: 80
subset: can
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: nginxdest
namespace: cdlg-edc-devci
spec:
host: nginx.cdlg-edc-devci.svc.cluster.local
subsets:
- name: prod
labels:
version: production
- name: can
labels:
version: canary
kind: Gateway
apiVersion: networking.istio.io/v1alpha3
metadata:
name: ccgf-gateway
namespace: namespace
spec:
servers:
- hosts:
- '*'
port:
name: http
number: 80
protocol: HTTP
apiVersion: v1
kind: Pod
metadata:
name: ubu
spec:
containers:
- name: ubu
image: ubuntu
command: ["/bin/sh"]
args: ["-c", "apt-get update && apt-get install curl -y && sleep 3000"]
Some results from ubuntu pod
curl -v external_istio-ingress_gateway_ip/cdlg-edc-devci/frontend
HTTP/1.1 200 OK
Hello nginx2
HTTP/1.1 200 OK
Hello nginx1
curl -v external_istio-ingress_gateway_ip/s
HTTP/1.1 200 OK
Hello nginx2
I hope it answer your question. Let me know if you have any more questions.

How to configure NGINX in AWS

I have set up k8s cluster in AWS ec2 instances with 1 parent and 2 child nodes using kops.
Deployed 2 services and running with LoadBalancer service type in browser.
Now I installed NGINX but through LB ip not able to hit my service. It is giving 504 GATEWAY_TIME_OUT exception. Googled it but no success, Where am I going wrong? Here is my sample code...[AWS FREE ACCOUNT]
deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: $APP_NAME
name: $APP_NAME
namespace: $NAMESPACE
spec:
replicas: 1
selector:
matchLabels:
app: $APP_NAME
template:
metadata:
labels:
app: $APP_NAME
spec:
imagePullSecrets:
- name: $IMG_PULL_SECRET
containers:
- image: $IMAGE_REG/$APP_NAME:$IMAGE_TAG
name: $APP_NAME
imagePullPolicy: Always
ports:
- containerPort: ${CONTAINER_PORT}
protocol: TCP
env:
- name: spring.cloud.config.uri
value: 'http://config-server-service'
service.yml
apiVersion: v1
kind: Service
metadata:
labels:
app: $APP_NAME
name: $APP_NAME
namespace: $NAMESPACE
spec:
type: $SERVICE_TYPE
#type: $SERVICE_TYPE
ports:
- port: 80
targetPort: ${CONTAINER_PORT}
protocol: TCP
selector:
app: $APP_NAME
ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ${APP_NAME}
namespace: $NAMESPACE
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/session-cookie-name: JSESSIONID
nginx.ingress.kubernetes.io/ssl-passthrough: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
kubernetes.io/ingress.class: "nginx"
kubernetes.io/ingress.allow-http: "true"
# kubernetes.io/ingress.global-static-ip-name: "my-gateway"
spec:
rules:
- http:
paths:
- path: /${APP_NAME}
backend:
serviceName: ${APP_NAME}
servicePort: 80

istio route rule error: "json: cannot unmarshal string into Go value"

I'm new to Istio and I'm going through some uses cases with my simple app.
I deployed 2 simple services on minikube running on Windows 10 Pro with VirtualBox 5.2.6, with istio 0.6.0
Version v1 of service1 and v1 & v2 of service2.
service1 responds to /hello and service2 responds to /world. Everything working fine so far and both services are responsding and as of service2, the round robin is working.
Now I want to apply 2 route rules, one to route to v2 of service2 based on a header and the rest to v1 of service2, but when I try to do that I get an error:
Error: cannot parse proto message: YAML decoding error: destination:
name: service2
match:
request:
headers:
Foo: bar
precedence: 2
route:
- labels:
version: v2
json: cannot unmarshal string into Go value of type map[string]json.RawMessage
Please find below my app and route rules config.
What's wrong with this config?
Please notice that when I omit the "match" part it's OK, but of course this is not what I want.
apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
name: service2-route
spec:
destination:
name: service2
precedence: 2
match:
request:
headers:
Foo: bar
route:
- labels:
version: v2
---
apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
name: service2-default
spec:
destination:
name: service2
precedence: 1
route:
- labels:
version: v1
weight: 100
and the my services deployment yaml:
###########################################################################
# Service 1
##########################################################################
apiVersion: v1
kind: Service
metadata:
name: service1
labels:
app: service1
spec:
ports:
- port: 8080
name: http
selector:
app: service1
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: service1-v1
spec:
replicas: 1
template:
metadata:
labels:
app: service1
version: v1
spec:
containers:
- name: service1
image: myrepo/sampleapp-service1:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
---
###########################################################################
# Service 2
##########################################################################
apiVersion: v1
kind: Service
metadata:
name: service2
labels:
app: service2
spec:
ports:
- port: 8081
name: http
selector:
app: service2
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: service2-v1
spec:
replicas: 1
template:
metadata:
labels:
app: service2
version: v1
spec:
containers:
- name: service2
image: myrepo/sampleapp-service2:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8081
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: service2-v2
spec:
replicas: 1
template:
metadata:
labels:
app: service2
version: v2
spec:
containers:
- name: service2
image: myrepo/sampleapp-service2:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8081
---
###########################################################################
# Ingress resource (gateway)
##########################################################################
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway
annotations:
kubernetes.io/ingress.class: "istio"
spec:
rules:
- http:
paths:
- path: /hello
backend:
serviceName: service1
servicePort: 8080
- path: /world
backend:
serviceName: service2
servicePort: 8081
---
The problem here is pretty simple, you have to say how do you want to match your header. In your example, I can assume that you want an exact match, so the following syntax:
match:
request:
headers:
Foo:
exact: bar
Here you can find more available options.
Also I would recommend to use quotes if your header value contains any special characters.