We have a cluster of several nodes so I can't do a NodePort and just go to my node-ip (which it's what I've done for testing prometheus).
I did a helm install stable/prometheus and stable/grafana at "monitoring" namespace.
Everything looks okay so far.
Then, I'm trying to create an LB service to access Grafana, which gets created, I can see the CNAME pointing to the A record for the ELB at AWS, but when accessing the URL of Grafana, nothing happens, no HTTP error, no problem page, nothing.
Here's the service-elb.yaml:
apiVersion: v1
kind: Service
metadata:
name: grafana-lb
namespace: monitoring
labels:
app: grafana
annotations:
dns.alpha.kubernetes.io/external: grafana-testing.country.ourdomain
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxxxxx
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '443'
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'
spec:
selector:
app: grafana
tier: frontend
type: LoadBalancer
ports:
- name: https
port: 443
targetPort: 80
- name: http
port: 80
targetPort: 3000
loadBalancerSourceRanges:
- somerange
- someotherrange
- etc etc
BTW, Got an error of permissions regarding serviceaccount if I don't create the chart with --set rbac.create=false
I recently use a nginx-proxy-pass for Kibana and also use a LB service similar to this with no issue. But I'm missing something here and can't find out what it is yet.
Any help will be much appreciated. I'll update if I make it work.
Solved, had to remove the "tier" selector and just use a spec like this:
spec:
selector:
app: grafana
type: LoadBalancer
ports:
- name: http
port: 3000
Related
I have an HTTP2 service. It's deployed on EKS (AWS Kubernetes). And I am trying to expose it to the internet.
If I am exposing it without TLS (with the code below) everything works fine. I can access it.
apiVersion: v1
kind: Service
metadata:
name: demoapp
spec:
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: 5000
selector:
name: demoapp
If I am adding TLS, I am getting HTTP 502 (Bad Gateway).
apiVersion: v1
kind: Service
metadata:
name: demoapp
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: somearn
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
spec:
type: LoadBalancer
ports:
- name: https
port: 443
targetPort: 5000
selector:
name: demoapp
I have a guess (which could be wrong) that service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http for reason assumes that it's HTTP 1.1 (vs HTTP 2.0) and bark when one of the sides start sending binary (vs textual data).
Additional note: I am not using any Ingress controller.
And a thought. Potentially, I can bring TLS termination within the app (vs doing it on the load balancer) and switch as an example to NLB. However, brings a lot of hair in the solution and I would rather use load balancer for it.
Base on the annotations in your question; the TLS should terminate at the CLB and you should remove service.beta.kubernetes.io/aws-load-balancer-backend-protocol (default to tcp).
I have a running private Kubernetes Cluster (v1.20) with fargate instances for the pods on the complete cluster. Access is restricted to the nodePorts range and 443.
I use externalDns to create Route53 entries to route from my internal network to the Kubernetes cluster. Everything works fine with e.g. UIs listening on port 443.
So now I want to use the kubernetes-dashboard not via proxy to my localhost but via DNS resolution with Route53. For this, I made two changes in the kubernetes-dashboard.yml:
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
is now:
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
annotations:
external-dns.alpha.kubernetes.io/hostname: kubernetes-dashboard.hostname.local
spec:
ports:
- port: 443
targetPort: 8443
externalTrafficPolicy: Local
type: NodePort
selector:
k8s-app: kubernetes-dashboard
and the container specs now contain the self-signed certificates:
spec:
containers:
- name: kubernetes-dashboard
image: kubernetesui/dashboard:v2.0.0
imagePullPolicy: Always
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
- --namespace=kubernetes-dashboard
- --tls-cert-file=/tls.crt
- --tls-key-file=/tls.key
The certificates (the same used for the UIs) are mapped via kubernetes secret.
The subnets for the fargate instances are the same as for my other applications. Yet i receive a "Connection refused" when i try to call my dashboard.
Checking the dashboard with the localproxy the configuration setup seems fine. Also the DNS entry is resolved to the correct IP address of the fargate instance.
My problem here is, that I already use this setup and it works. But I see no difference to the dashboard here. Do I miss something?
Can anybody help me out here?
Greetings,
Eric
I am trying to implement SonarQube in a Kubernetes cluster. The deployment is running properly and is also exposed via a Virtual Service. I am able to open the UI via the localhost:port/sonar but I am not able to access it through my external ip. I understand that sonar binds to localhost and does not allow access from outside the remote server. I am running this on GKE with a MYSQL database. Here is my YAML file:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sonarqube
namespace: sonar
labels:
service: sonarqube
version: v1
spec:
replicas: 1
template:
metadata:
name: sonarqube
labels:
name: sonarqube
spec:
terminationGracePeriodSeconds: 15
initContainers:
- name: volume-permission
image: busybox
command:
- sh
- -c
- sysctl -w vm.max_map_count=262144
securityContext:
privileged: true
containers:
- name: sonarqube
image: sonarqube:6.7
resources:
limits:
memory: 4Gi
cpu: 2
requests:
memory: 2Gi
cpu: 1
args:
- -Dsonar.web.context=/sonar
- -Dsonar.web.host=0.0.0.0
env:
- name: SONARQUBE_JDBC_USERNAME
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: username
- name: SONARQUBE_JDBC_PASSWORD
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: password
- name: SONARQUBE_JDBC_URL
value: jdbc:mysql://***.***.**.*:3306/sonar?useUnicode=true&characterEncoding=utf8
ports:
- containerPort: 9000
name: sonarqube-port
---
apiVersion: v1
kind: Service
metadata:
labels:
service: sonarqube
version: v1
name: sonarqube
namespace: sonar
spec:
selector:
name: sonarqube
ports:
- name: http
port: 80
targetPort: sonarqube-port
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: sonarqube-internal
namespace: sonar
spec:
hosts:
- sonarqube.staging.jeet11.internal
- sonarqube
gateways:
- default/ilb-gateway
- mesh
http:
- route:
- destination:
host: sonarqube
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: sonarqube-external
namespace: sonar
spec:
hosts:
- sonarqube.staging.jeet11.com
gateways:
- default/elb-gateway
http:
- route:
- destination:
host: sonarqube
---
The deployment completes successfully. My exposed services gives a public ip that has been mapped to the host url but I am unable to access the service at the host url.
I need to change the mapping such that sonar binds with the server ip but I am unable to understand how to do that. I cannot bind it to my cluster ip, neither to my internal or external service ip.
What should I do? Please help!
I had the same issue recently and I managed to get this resolved today.
I hope the following solution will work for anyone facing the same issue!.
Environment
Cloud Provider: Azure - AKS
This should work regardless of whatever provider you use.
Istio Version: 1.7.3
K8 Version: 1.16.10
Tools - Debugging
kubectl logs -n istio-system -l app=istiod
logs from Istiod and events happening in the control plane.
istioctl analyze -n <namespace>
This generally gives you any warnings and errors for a given namespace.
Lets you know if things are misconfigured.
Kiali - istioctl dashboard kiali
See if you are getting inbound traffic.
Also, shows you any misconfigurations.
Prometheus - istioctl dashboard prometheus
query metric - istio_requests_total. This shows you the traffic going into the service.
If there's any misconfiguration you will see the destination_app as unknown.
Issue
Unable to access sonarqube UI via external IP, but accessible via localhost (port-forward).
Unable to route traffic via Istio Ingressgateway.
Solution
Sonarqube Service Manifest
apiVersion: v1
kind: Service
metadata:
name: sonarqube
namespace: sonarqube
labels:
name: sonarqube
spec:
type: ClusterIP
ports:
- name: http
port: 9000
targetPort: 9000
selector:
app: sonarqube
status:
loadBalancer: {}
Your targetport is the container port. To avoid any confusion just assign the service port number as same as the service targetport.
The port name is very important here. “Istio required the service ports to follow the naming form of ‘protocol-suffix’ where the ‘-suffix’ part is optional” - KIA0601 - Port name must follow [-suffix] form
Istio Gateway and VirtualService manifest for sonarqube
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: sonarqube-gateway
namespace: sonarqube
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 9000
name: http
protocol: HTTP
hosts:
- "XXXX.XXXX.com.au"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: sonarqube
namespace: sonarqube
spec:
hosts:
- "XXXX.XXXX.com.au"
gateways:
- sonarqube-gateway
http:
- route:
- destination:
host: sonarqube
port:
number: 9000
Gateway protocol must be set to HTTP.
Gateway Server Port and VirtualService Destination Port is the same. If you have different app Service Port, then your VirtualService Destination Port number should match the app Service Port. The Gateway Server Port should match the app Service Targetport.
Now comes to the fun bit! The hosts. If you want to access the service outside of the cluster, then you need to have your host-name (whatever host-name that you want to map the sonarqube server) as an DNS A record mapped to the External Public IP address of the istio-ingressgateway.
To get the EXTERNAL-IP address of the ingressgateway, run kubectl -n istio-system get service istio-ingressgateway.
If you do a simple nslookup (run - nslookup <hostname>), The IP address you get must match with the IP address that is assigned to the istio-ingressgateway service.
Expose a new port in the ingressgateway
Note that your sonarqube gateway port is a new port that you are introducing to Kubernetes and you’re telling the cluster to listen on that port. But your load balancer doesn’t know about this port. Therefore, you need to open the specified gateway port on your kubernetes external load balancer. Ref - Info
You don’t need to manually change your load balancer service. You just need to update the ingress gateway to include the new port, which will update the load balancer automatically.
You can identify if the port is causing issues by running istioctl analyze -n sonarqube. You should get the following warning;
[33mWarn[0m [IST0104] (Gateway sonarqube-gateway.sonarqube) The gateway refers to a port that is not exposed on the workload (pod selector istio=ingressgateway; port 9000) Error: Analyzers found issues when analyzing namespace: sonarqube. See https://istio.io/docs/reference/config/analysis for more information about causes and resolutions.
You should get the corresponding error in the control plane. Run kubectl logs -n istio-system -l app=istiod.
At this point you need to update the Istio ingressgateway service to expose the new port. Run kubectl edit svc istio-ingressgateway -n istio-system and add the following section to the ports.
Bypass creating a new port
In the previous section you saw how to expose a new port. This is optional and depending on your use case.
In this section you will see how to use a port that is already exposed.
If you look at the service of the istio-ingressgateway. You can see that there are default ports exposed. Here we are going to use port 80.
Your setup will look like the following;
To void specifying the port with your host name just add match uri prefix, as shown in the virtualservice manifest.
Time for testing
If everything works up to this point as expected, then you are good to go.
During testing I made one mistake by not specifying the port. If you get 404 status, Which is still a good thing, in this way you can verify what server it is using. If you setup things correctly, it should use the istio-envoy server, not the nginx.
Without specifiying the port. This will only work if you add the match uri prefix.
Donot pass argument just try running without it once working for me.
This is how my deployment file hope helpful
apiVersion: v1
kind: Service
metadata:
name: sonarqube-service
spec:
selector:
app: sonarqube
ports:
- protocol: TCP
port: 9000
targetPort: 9000
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: sonarqube
name: sonarqube
spec:
replicas: 1
template:
metadata:
labels:
app: sonarqube
spec:
containers:
- name: sonarqube
image: sonarqube:7.1
resources:
requests:
memory: "1200Mi"
cpu: .10
limits:
memory: "2500Mi"
cpu: .50
volumeMounts:
- mountPath: "/opt/sonarqube/data/"
name: sonar-data
- mountPath: "/opt/sonarqube/extensions/"
name: sonar-extensions
env:
- name: "SONARQUBE_JDBC_USERNAME"
value: "root" #Put your db username
- name: "SONARQUBE_JDBC_URL"
value: "jdbc:mysql://192.168.112.4:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true" #DB URL
- name: "SONARQUBE_JDBC_PASSWORD"
value : password
ports:
- containerPort: 9000
protocol: TCP
volumes:
- name: sonar-data
persistentVolumeClaim:
claimName: sonar-data
- name: sonar-extensions
persistentVolumeClaim:
claimName: sonar-extensions
as far as I get the ingress controller documentation, a simple creation of a Service and an Ingress without special annotations should create internet-facing load balancers, weirdly it is creating internal load balancers. So I added the annotation service.beta.kubernetes.io/aws-load-balancer-internal: "false" which is not working either. By the way, I am using NGINX as ingress controller, in the test cluster currently in version 0.8.21. Probably I should update it some time.
Here's my simple spec-file:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: nginx
service.beta.kubernetes.io/aws-load-balancer-internal: "false"
labels:
external: "true"
comp: ingress-nginx
env: develop
name: develop-api-external-ing
namespace: develop
spec:
rules:
- host: api.example.com
http:
paths:
- backend:
serviceName: api-external
servicePort: 3000
path: /
tls:
- hosts:
- api.example.com
secretName: api-tls
---
apiVersion: v1
kind: Service
metadata:
labels:
app: api
env: develop
name: api-external
namespace: develop
spec:
ports:
- name: http
port: 3000
protocol: TCP
targetPort: 3000
selector:
app: api
env: develop
sessionAffinity: None
type: ClusterIP
You are not wrong, a service and a ingress should create a load balancer... but you should look at the documentation a bit more...
An ingress needs a NodePort service, yours is ClusterIP. So even if it created something it wouldn't work.
In your ingress you are using kubernetes.io/ingress.class: nginx meaning you want to override the default usage of the ingress and force it to register to the ingress-nginx.
So to make it work, change the type of your service, remove the ingress-class annotation.
You can setup NLB ( Network load balancer) and provide the URL on ingress rule host values. You don't need to expose the underneath backend service either as NodePort or as another load balancer.
I have a LoadBalancer service on a k8s deployment on aws (made via kops).
Service definition is as follows:
apiVersion: v1
kind: Service
metadata:
name: ui
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <certificate_id>
spec:
ports:
- name: http
port: 80
targetPort: ui-port
protocol: TCP
- name: https
port: 443
targetPort: ui-port
protocol: TCP
selector:
els-pod: ui
type: LoadBalancer
Here is the respective deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ui-deployment
spec:
replicas: 1
template:
metadata:
labels:
els-pod: ui
spec:
containers:
- image: <my_ecr_registry>/<my_image>:latest
name: ui
ports:
- name: ui-port
containerPort: 80
restartPolicy: Always
I know that <my_image> exposes port 80.
I have also assigned an alias to the ELB that gets deployed, say. my-k8s.mydomain.org
Here is the issue:
https://my-k8s.mydomain.org works just fine
http://my-k8s.mydomain.org returns an empty page (when accessing behind a squid proxy, I get the zero-sized reply error message)
Why am I unable to access the service via port 80?
edit: what I have just found is that the service annotation regarding the certificate, also assigns it on port 80 on the ELB.
Could that be the issue?
Is there a way around this?
Just needed to add the following annotation in the service definition:
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"