A newly created Kubernetes cluster on GKE is not pushing its metrics to Stackdriver. Output of kubectl cluster-info is:
Kubernetes master is running at https://XXX.XXX.XXX.XXX
KubeDNS is running at https://XXX.XXX.XXX.XXX/api/v1/proxy/namespaces/kube-system/services/kube-dns
KubeUI is running at https://XXX.XXX.XXX.XXX/api/v1/proxy/namespaces/kube-system/services/kube-ui
Heapster is running at https://XXX.XXX.XXX.XXX/api/v1/proxy/namespaces/kube-system/services/monitoring-heapster
When I try to create a dashboard on Stackdriver with 'Custom Metrics', it says 'No Match Found'. Metrics were supposed to be present at this location with 'kubernetes.io' prefix according to Heapster documentation.
I have also enabled Cloud Monitoring API with Read Write permission while creating cluster. Is it required for pushing cluster metrics?
What Heapster does with the metrics depends on its configuration. When running as part of GKE, the metrics aren't exported as "custom" metrics, but rather as official GKE service metrics. The feature is still in an experimental, soft-launch state, but you should be able to access them at app.google.stackdriver.com/gke
In the documentation it says you must enable monitoring by running:
gcloud alpha container clusters update --monitoring-service=monitoring.googleapis.com <cluster-name>
This is supposed to be on by default but it wasn't for me.
Related
I've created a RabbitMQ kubernetes cluster using Google One Click to deploy. I've checked "Enable Stackdriver Metrics Exporter" and created the cluster. My problem is that Google is charging for every custom metric created.
I need to disable Stackdriver Metrics Exporter.
¿Anyone had the same issue and disabled this Exporter? If so ¿How can I disable it without destroying the cluster?
If this kubernetes cluster without another application, only RabbitMQ is running on it, you can disable “Stackdriver Kubernetes Engine Monitoring” function of kubernetes cluster.
In the Cloud Console, go to the Kubernetes Engine > Kubernetes clusters page:
Click your cluster.
Click Edit for the cluster you want to change.
Set the “Stackdriver Kubernetes Engine Monitoring” drop-down value to Disabled.
Click Save.
The Logs ingestion page in the Logs Viewer tracks the volume of logs in your project. The Logs Viewer also gives you tools to disable all logs ingestion or exclude (discard) log entries you're not interested in, so that you can minimize any charges for logs over your monthly allotment.
Go to logs exports, and follow this topic for manage "Logs Exports".
I use Google Kubernetes Engine(GKE) to deploy my service. In the cluster, I enable Stackdriver Kubernetes Engine Monitoring instead of Legacy Stackdriver Logging and Legacy Stackdriver Monitoring. With the legacy monitor, I can find the metrics of the number of logs with the name log entries. What is the corresponding metrics name with Stackdriver Kubernetes Engine Monitoring?
If you go to Stackdriver monitoring > Resources > Metrics Explorer and select "Kubernetes cluster" as a resource type, you can find a metric called "log_entry_count" and select it. This metric is also mentioned here.
So - the metric you're asking about is still there - no matter if you create a cluster with Stackdriver Kubernetes Engine Monitoring enabled or no.
Furthermore - it will still collect data about number of logs ingested.
To be sure of the metric existence and if it actually does work I created a test cluster with some back-end service which generated some log entries and then tried "log entries" metric to count them - it worked as it should.
Following the steps provided in this documentation.
I was looking into better monitoring of our GKE cluster and so thought I'd try out the beta kubernetes Stackdriver monitoring. My cluster version is 1.11.7 (later than the suggested 1.11.2) and I created the cluster with the --enable-stackdriver-kubernetes flag.
In the cluster details Stackdriver logging and monitoring is listed as 'Enabled v2(beta)' however in the stackdriver resources menu the 'kubernetes beta' option will simply not appear as shown here.
I have also confirmed fluentd, heapster and metadata-agent pods are running within the cluster as suggested by the docs.
Any possible suggestions are much appreciated.
I managed to resolve this issue:
Firstly the 'Kubernetes Beta' option appeared in Stackdriver appeared without me making any changes to the cluster(Slightly annoying)
I gave the clusters service account the appropriate monitoring and logging roles.
This happens while trying to create a VPC-native GKE cluster. Per the documentation here the command to do this is
gcloud container clusters create [CLUSTER_NAME] --enable-ip-alias
However this command, gives below error.
ERROR: (gcloud.container.clusters.create) Only alpha clusters (--enable_kubernetes_alpha) can use --enable-ip-alias
The command does work when option --enable_kubernetes_alpha is added. But gives another message.
This will create a cluster with all Kubernetes Alpha features enabled.
- This cluster will not be covered by the Container Engine SLA and
should not be used for production workloads.
- You will not be able to upgrade the master or nodes.
- The cluster will be deleted after 30 days.
Edit: The test was done in zone asia-south1-c
My questions are:
Is VPC-Native cluster production ready?
If yes, what is the correct way to create a production ready cluster?
If VPC-Native cluster is not production ready, what is the way to connect privately from a GKE cluster to another GCP service (like Cloud SQL)?
Your command seems correct. Seems like something is going wrong during the creation of your cluster on your project. Are you using any other flags than the command you posted?
When I set my Google cloud shell to region europe-west1
The cluster deploys error free and 1.11.6-gke.2(default) is what it uses.
You could try to manually create the cluster using the GUI instead of gcloud command. While creating the cluster, check the “Enable VPC-native (using alias ip)” feature. Try using a newest non-alpha version of GKE if some are showing up for you.
Public documentation you posted on GKE IP-aliasing and the GKE projects.locations.clusters API shows this to be in GA. All signs point this to be production ready. For whatever it’s worth, the feature has been posted last May In Google Cloud blog.
What you can try is to update your version of Google Cloud SDK. This will bring everything up to the latest release and remove alpha messages for features that are in GA right now.
$ gcloud components update
I am running a Kubernetes cluster on Google container engine. My metrics are not getting pushed to Stackdriver by default.
Do I need to start Heapster service explicitly or is it automatically managed by container engine itself?
If there isn't a Heapster pod running in your cluster, then your cluster was created before we started enabling cluster monitoring by default.
We are working on adding a way for users to retroactively turn on monitoring, but if you want metrics pushed into stack driver today you will need to create a new cluster (launching Heapster yourself isn't sufficient for the metrics to get collected).