How to disable google load balancer logging? We are currently generating 30TB (Yes TB not GB) of logs per month that we don't care about and never look at. It's quite a waste or resources.
In Google Cloud Platform console, from Stackdriver Logging section, navigate to Resource usage, choose Cloud HTTP Load Balancer and select Disable log source from the options on the right. https://console.cloud.google.com/logs/usage
You can also see option to create Exclusion filter.
Reference: https://cloud.google.com/logging/docs/exclusions#excluding-resource
To delete the previous logs
gcloud logging logs delete projects/<PROJECT-ID>/logs/requests
As per the Google Cloud Load Balancer and Stackdriver Logging documentation and also going over the Cloud Console UI and the REST API documentation, there is currently no way to disable Stackdriver logging for Google Cloud Load balancer.
Also, Google currently does not charge a fee for storing the Stackdriver logs.
If your concern is that it is hard to search through the logs, you can always filter logs while viewing on Stackdriver to help you narrow down only the items of interest.
Related
I wrote some code to automate the training procedure on our company vm instances.
you probably know that sometimes GCP can't provide you at the current moment with a machine - 'out of resource' exception.
so , I'd like to monitor which of my machines successfully turned on and which not.
if there is some way to show it on Bigquery it will be great.
thanks .
Using the Cloud Monitoring (Stackdriver) functionality is good way for monitoring all you VMs.
Here is a detailed guide to implement Monitoring on a Compute Engine Instance.
Hope you find it useful.
You can use Google cloud's activity logs too:
Activity logging is enabled by default for all Compute Engine
projects.
You can see your project's activity logs through the Logs Viewer in
the Google Cloud Console:
In the Cloud Console, go to the Logging page. Go to the Logging page
When in the Logs Viewer, select and filter your resource type from the
first drop-down list. From the All logs drop-down list, select
compute.googleapis.com/activity_log to see Compute Engine activity
logs.
Here is the Official documentation.
Does Metric Registrar works in Cloud Foundry without Pivotal?
I have open source Cloud Foundry and I need to get custom metrics from app. I installed Metric Registrar community plugin for CF, I registered my application with endpoint, I also defined log format. Unfortunately I see no traffic on registered endpoint.
If open source Cloud Foundry do not support Metric Registrar, is there any other way to get support for custom app metrics?
Does Metric Registrar works in Cloud Foundry without Pivotal?
The Metric Registrar is part of the VMware Tanzu Application Service product, it's not part of the Open Source Cloud Foundry project. It's a value-add feature for those using the paid product.
If open source Cloud Foundry do not support Metric Registrar, is there any other way to get support for custom app metrics?
You don't strictly need the Metric Registrar to do this. The Metric Registrar's main purpose is to take metrics from your apps and inject them into the Loggregator log/metric stream. This is convenient if you have other software that is already consuming log & metric streams from Loggregator.
You don't have to do that though, as there are other ways to export metrics from your app.
If you want them to go through Loggregator, you could export structured log messages (perhaps JSON?) via STDOUT that contains your metrics. Those will, like your other log messages, go out through Loggregator. You would then just need to have something ingesting your logs, identifying the structured messages, and parsing out your metrics. This is similar to what Metric Registrar does, you're just parsing out the structured log entries after they leave the platform.
If you have an ELK stack or similar running, you can probably make this solution work easily enough. ELK can ingest your logs & structured log metrics, then you can search/filter through the metrics and create dashboards.
Another option you could do is to run Prometheus/Grafana. You then just need to make sure your app has a Prometheus Exposition metrics endpoint (this is super easy with Java/Spring Boot & Spring Boot Actuator, but can be done in any language). Point Prometheus at your app and it will then be able to scrape metrics from your apps & you can use Grafana to view them. None of this goes through Loggregator.
If you're looking for a solution that's more automatic, you could run an APM agent (NewRelic, DataDog, AppDynamics, Dynatrace, etc..) with your apps. These will capture metrics directly from the process and export them to a SaaS platform where you can monitor/review them.
There are probably other options as well. This is just what comes to mind as I write this up.
I've created a RabbitMQ kubernetes cluster using Google One Click to deploy. I've checked "Enable Stackdriver Metrics Exporter" and created the cluster. My problem is that Google is charging for every custom metric created.
I need to disable Stackdriver Metrics Exporter.
¿Anyone had the same issue and disabled this Exporter? If so ¿How can I disable it without destroying the cluster?
If this kubernetes cluster without another application, only RabbitMQ is running on it, you can disable “Stackdriver Kubernetes Engine Monitoring” function of kubernetes cluster.
In the Cloud Console, go to the Kubernetes Engine > Kubernetes clusters page:
Click your cluster.
Click Edit for the cluster you want to change.
Set the “Stackdriver Kubernetes Engine Monitoring” drop-down value to Disabled.
Click Save.
The Logs ingestion page in the Logs Viewer tracks the volume of logs in your project. The Logs Viewer also gives you tools to disable all logs ingestion or exclude (discard) log entries you're not interested in, so that you can minimize any charges for logs over your monthly allotment.
Go to logs exports, and follow this topic for manage "Logs Exports".
I use Google Kubernetes Engine(GKE) to deploy my service. In the cluster, I enable Stackdriver Kubernetes Engine Monitoring instead of Legacy Stackdriver Logging and Legacy Stackdriver Monitoring. With the legacy monitor, I can find the metrics of the number of logs with the name log entries. What is the corresponding metrics name with Stackdriver Kubernetes Engine Monitoring?
If you go to Stackdriver monitoring > Resources > Metrics Explorer and select "Kubernetes cluster" as a resource type, you can find a metric called "log_entry_count" and select it. This metric is also mentioned here.
So - the metric you're asking about is still there - no matter if you create a cluster with Stackdriver Kubernetes Engine Monitoring enabled or no.
Furthermore - it will still collect data about number of logs ingested.
To be sure of the metric existence and if it actually does work I created a test cluster with some back-end service which generated some log entries and then tried "log entries" metric to count them - it worked as it should.
I have setup a Managed Instance Group with initial 3 instances (I installed Lumen inside, and the web server is auto started) to be used with the GCP load balancer. The LB works great.
However, whenever I need to trace lumen logs, I need to SSH every single instance to view the logs. Is there any best practices of one centralized storage I can refer to for the logs?
Can I mount the lumen logs into a centralized disk e.g. GCP filestore volume, or Google storage bucket or using FluendD to dump my logs into GCP Logging?
Please, I need to know the best industrial practice. THanks
STACK DRIVER is the right option for your case
https://cloud.google.com/logging/docs/agent/installation#joint-install
Install the stack driver logging agent on compute engine instances. You can track your logs lively also you can create visualizations and useful analysis out of it. Stackdriver is the best industry standard for the people who is using GCP. remember the pricing. Please check the pricing details