I've deployed successfully an app to K8s with istio
We have gw which we use and virtual service like the following:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bher-virtualservice
namespace: ba-trail
spec:
gateways:
- bher-gateway
hosts:
- trialio.cloud.str
http:
- match:
- uri:
prefix: "/"
- uri:
prefix: "/login"
- uri:
prefix: "/static"
- uri:
regex: '^.*\.(ico|png|jpg)$'
route:
- destination:
host: bsa.ba-trail.svc.cluster.local service.namespace.svc.cluster.local
port:
number: 5000
I defined also a service and deployment.
I want to expose the service outside that I will be able to access
like:
https://myapp.host:5000
when I run:
kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 100.61.114.202 a7151b2063cb-200880.eu-central-1.elb.amazonaws.com 150210:31161/TCP,80:31280/TCP,443:31190/TCP 41d
How it can be done?
I was able to run the app with port forwarding but I want a direct external link.
So in your case, you have an ELB serving your istio ingress gateway that goes to a VirtualService that directs traffic to port 5000 in the container.
I assume that you have it working with 🤔💭:
a7151b2063cb-200880.eu-central-1.elb.amazonaws.com:80 and
a7151b2063cb-200880.eu-central-1.elb.amazonaws.com:443 ❓
and you want something like:
a7151b2063cb-200880.eu-central-1.elb.amazonaws.com:5000 ❓
but with a specific name that maps to
myapp.host ❓
First, you have to create a DNS CNAME record that maps myapp.host to a7151b2063cb-200880.eu-central-1.elb.amazonaws.com.
Then on the Kubernetes service istio-ingressgateway you probably have something like this:
apiVersion: v1
kind: Service
metadata:
name: istio-ingressgateway
namespace: istio-system
labels:
name: istio-ingress-service
annotations:
... (❓)
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: ❓
protocol: TCP
- port: 443
targetPort: ❓
protocol: TCP
selector:
name: something-that-matches-your-istio-ingress
You could just add the extra port to the service so that it listens on that port on the outside.
apiVersion: v1
kind: Service
metadata:
name: istio-ingressgateway
namespace: istio-system
labels:
name: istio-ingress-service
annotations:
... (❓)
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: ❓
protocol: TCP
- port: 443
targetPort: ❓
protocol: TCP
- port: 5000
targetPort: ❓
selector:
name: something-that-matches-your-istio-ingress
Finally, the virtual service needs to match your hostname myapp.host
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bher-virtualservice
namespace: ba-trail
spec:
gateways:
- bher-gateway
hosts:
- myapp.host
...
✌️
Related
Ingress is not routing traffic to backend service port. Connection times out.
Any idea how to troubleshoot? There are no errors in the ingress controller logs.
curl -v -L https://www.example.com
* Trying 172.20.xxx.xx:443...
I've checked the Loadbalancer and its health is fine. I'm not sure why its not routing traffic to backend port and what else I could check to figure out what the issue is.
Here is my aws ingress controller.yaml config :
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:xxx"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.3.0
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: http
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: LoadBalancer
And here the service config service.yaml:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2022-08-06T12:16:17Z"
labels:
app: manageiq
name: httpd
namespace: default
spec:
clusterIP: 100.69.xxx.xx
clusterIPs:
- 100.69.xxx.xx
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
name: httpd
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
Ingress.yaml:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
creationTimestamp: "2022-08-06T12:47:32Z"
generation: 2
labels:
app: manageiq
name: httpd
namespace: default
spec:
rules:
- host: www.example.com
http:
paths:
- backend:
service:
name: httpd
port:
number: 8080
path: /
pathType: ImplementationSpecific
status:
loadBalancer:
ingress:
- hostname: adc1d30f0db264d7ea54aed4dcdc12ec-atest.elb.ap-south-1.amazonaws.com
Figured it out. Loadbalancer was set to internal. I was following a documentation.
service.beta.kubernetes.io/aws-load-balancer-internal: "false"
I had to delete services and recreate so a new load balancer gets created.
I want to secure my web service running on Kubernetes (EKS). It is running on port 80 .I want to run this on port 443.
When I apply the YAML file (for service and ingress), on AWS console I still have it listening on port 80 (and not on 443):
This is my YAML file:
How can I let it works? Thanks for you time!
#SERVICE LOGGER
apiVersion: v1
kind: Service
metadata:
name: load-balancer-api-logger
namespace: servicename-core-ns
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:786543355018:certificate/acdff29d4-7a32-42f1-8f11-1d4f495a5c77
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/force-ssl-redirect: "true"
spec:
selector:
app: api-logger
type: NodePort
ports:
- protocol: TCP
port: 443
targetPort: 5000
selector:
app.kubernetes.io/name: api-logger
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-articor
namespace: servicename-core-ns
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/healthcheck-path: "/healthcheckep"
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
alb.ingress.kubernetes.io/ssl-redirect: '443'
spec:
rules:
- host: logger.domainname.com
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: load-balancer-api-logger
port:
number: 80
Please consider that if I try to manually set the ALB to work with HTTPS it works fine. What I'm trying to achive here is to configure it via YAML file.
You should configure all settings in Ingress object. The following spec also don't repeat the default value set by the controller:
apiVersion: v1
kind: Service
metadata:
name: load-balancer-api-logger
namespace: servicename-core-ns
spec:
selector:
app: api-logger
type: NodePort
ports:
- protocol: TCP
port: 443
targetPort: 5000
selector:
app.kubernetes.io/name: api-logger
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-articor
namespace: servicename-core-ns
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/healthcheck-path: "/healthcheckep"
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-central-1:786543355018:certificate/acdff29d4-7a32-42f1-8f11-1d4f495a5c77
spec:
rules:
- host: logger.domainname.com
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: load-balancer-api-logger
port:
number: 443
I have created two ingresses, one for Grafana and one for my App. When the external dns write them into the route53 hosted zone as a A record, only one of them (the Myapp dns) get the (E)LB alias (dns), though the second A record get the internal IP as an ip address in the route53 A record.
the big question:
is there a way to set them all to the same alias/or at list to the same elb?
why it doesn't success doing so by default?
using:
terraform:
helm_release:
nginx-controller-bitnami
external-dns-bitnami
prometheus community
grafana-ingress:
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
meta.helm.sh/release-name: kube-prometheus-stack
meta.helm.sh/release-namespace: prometheus
generation: 1
labels:
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: grafana
app.kubernetes.io/version: 8.3.3
helm.sh/chart: grafana-6.20.5
name: kube-prometheus-stack-grafana
namespace: prometheus
resourceVersion: "2419"
spec:
rules:
- host: grafana.dns.io
http:
paths:
- backend:
service:
name: kube-prometheus-stack-grafana
port:
number: 80
path: /
pathType: Prefix
status:
loadBalancer:
ingress:
- ip: 10.0.1.19
kind: List
metadata:
resourceVersion: ""
selfLink: ""
my-app-ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
annotations:
ingress.kubernetes.io/ssl-redirect: /
nginx.ingress.kubernetes.io/rewrite-target: /$1
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: "some.website.io"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: "{{ .Release.Name }}-app"
port:
number: 80
status:
loadBalancer:
ingress:
- ip: 10.0.1.19
nginx-controller-service
kind: Service
metadata:
annotations:
external-dns.alpha.kubernetes.io/hostname: website.my-app.io, some.grafana.io
meta.helm.sh/release-name: nginx-ingress-controller
meta.helm.sh/release-namespace: default
finalizers:
- service.kubernetes.io/load-balancer-cleanup
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: nginx-ingress-controller
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx-ingress-controller
helm.sh/chart: nginx-ingress-controller-9.1.4
name: nginx-ingress-controller
namespace: default
resourceVersion: "997"
spec:
clusterIP: 172.30.0.140
clusterIPs:
- 172.30.0.140
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
nodePort: 30337
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 31512
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: nginx-ingress-controller
app.kubernetes.io/name: nginx-ingress-controller
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- hostname: someElbDns.eu-west-1.elb.amazonaws.com
Since your ingress controller service is of type LoadBalancer, creating this service will provision an NLB for you which will have your nginx controller pods as target groups.
A request will pass from the NLB to the ingress pods and then to grafana/your app subsequently.
I would try to use the same host (website.io) and the same nginx ingress class so both ingresses get assigned the same NLB address:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: default
name: grafana-ingress
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- grafana.website.io
secretName: tls-secret-website-io
rules:
- host: website.io
- http:
paths:
- path: /
backend:
serviceName: grafana-service
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: default
name: awebsite-ingress
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- subdomain.website.io
secretName: tls-secret-website-io
rules:
- host: website.io
- http:
paths:
- path: /(/|$)(.*)
backend:
serviceName: website-service
servicePort: 80
kind: Service
apiVersion: v1
metadata:
name: nginx-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: nginx-controller
app.kubernetes.io/part-of: nginx-controller
annotations:
# by default the type is elb (classic load balancer).
service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
# this setting is to make sure the source IP address is preserved.
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: nginx-controller
app.kubernetes.io/part-of: nginx-controller
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
Last step is to create a CNAME entry in Rote53 from your domain to the loadbalancer DNS.
I've create loadbalancer for my microservices, with this template:, all is good and works but wanted to somehow add the global flag (when you create lb through gcp console you have option to add it) to meet expectations of the app functionality, does anyone know what other flag might I need to add ?
apiVersion: v1
kind: Service
metadata:
name: my-app-jmprlb
annotations:
cloud.google.com/load-balancer-type: "Internal"
labels:
app: my-app
env: dev
spec:
type: LoadBalancer
selector:
app: my-app
env: dev
ports:
- port: 80
targetPort: 8080
protocol: TCP
loadBalancerIP: 10.10.10.10
externalTrafficPolicy: Local
EDIT:
I found some nice annotations from google docs, seem to do the trick,https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balance-ingress
# web-service.yaml
apiVersion: v1
kind: Service
metadata:
name: hostname
namespace: default
annotations:
cloud.google.com/neg: '{"ingress": true}'
spec:
ports:
- name: host1
port: 80
protocol: TCP
targetPort: 9376
selector:
app: hostname
type: NodePort
and
# internal-ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ilb-demo-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: "gce-internal"
spec:
backend:
serviceName: hostname
servicePort: 80
If you want to make it a global LoadBalancer which accessible from the outside your cluster with public IP you can use:
apiVersion: v1
kind: Service
metadata:
name: my-app-jmprlb
labels:
app: my-app
env: dev
spec:
type: LoadBalancer
selector:
app: my-app
env: dev
ports:
- port: 80
targetPort: 8080
protocol: TCP
Note that the annotation of cloud.google.com/load-balancer-type: "Internal" means that your service is only accessible withing subnets that were peer with the subnet where your cluster resided.
I have a GKE cluster, a static ip, and a container/pod which exposes 2 ports: 8081(UI https) and 8082 (WSS https). I must connect to the "UI Https" and "WSS Https" on the same IP. The "WSS Https" service does not have a health check endpoint.
Do i need to use Isito, Consul, Nginx ingress or some service mesh to allow these connections on the same IP with different ports?
Is this even possible?
Things i have tried:
GCP global lb with 2 independent ingress services. The yaml for the second service never works correctly but i can add another backed service via the UI. The ingress always reverts to the default health check for the "WSS Https" service and it always unhealthy.
Changed Service type from NodePort to LoadBalancer with a static ip. This will not allow me to change the readiness check and always reverts back.
GCP GLIB with 1 ingress and 2 backend gives me the same healthcheck failure as above
TCP Proxy - Does not allow me to set the same instance group.
Below are my ingress, service, and deployment.
Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: app-ingress
namespace: myappnamespace
annotations:
kubernetes.io/ingress.global-static-ip-name: global-static-ip-name
labels:
app: appname
spec:
backend:
serviceName: ui-service
servicePort: 8081
tls:
- hosts:
- my-host-name.com
secretName: my-secret
rules:
- host: my-host-name.com
http:
paths:
- backend:
serviceName: ui-service
servicePort: 8081
- host: my-host-name.com
http:
paths:
- backend:
serviceName: app-service
servicePort: 8082
Services
---
apiVersion: v1
kind: Service
metadata:
labels:
name: ui-service
name: ui-service
namespace: myappnamespace
annotations:
cloud.google.com/app-protocols: '{"ui-https":"HTTPS"}'
beta.cloud.google.com/backend-config: '{"ports":{"8081":"cloud-armor"}}'
spec:
selector:
app: appname
ports:
- name: ui-https
port: 8081
targetPort: "ui"
protocol: "TCP"
selector:
name: appname
type: NodePort
---
apiVersion: v1
kind: Service
metadata:
labels:
name: app-service
name: app-service
namespace: myappnamespace
annotations:
cloud.google.com/app-protocols: '{"serviceport-https":"HTTPS"}'
beta.cloud.google.com/backend-config: '{"ports":{"8082":"cloud-armor"}}'
spec:
selector:
app: appname
ports:
- name: serviceport-https
port: 8082
targetPort: "service-port"
protocol: "TCP"
selector:
name: appname
type: NodePort
---
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: appname
namespace: myappnamespace
labels:
name: appname
spec:
replicas:1
selector:
matchLabels:
name: appname
strategy:
type: Recreate
template:
metadata:
name: appname
namespace: appnamespace
labels:
name: appname
spec:
restartPolicy: Always
serviceAccountName: myserviceaccount
containers:
- name: my-container
image: image
ports:
- name: service-port
containerPort: 8082
- name: ui
containerPort: 8081
readinessProbe:
failureThreshold: 3
httpGet:
path: /api/health
port: 8081
scheme: HTTPS
livenessProbe:
exec:
command:
- cat
- /version.txt
[......]
A Service exposed through an Ingress must respond to health checks from the load balancer.
External HTTP(S) Load Balancer that GKE Ingress creates only supports port 443 for https traffic.
In that case you may want to:
Use two separate Ingress resources to route traffic for two different host names on the same IP address and port:
ui-https.my-host-name.com
wss-https.my-host-name.com
Opt to use Ambassador or Istio Virtual Service.
Try Multi-Port Services.
Please let me know if that helped.