Configured Google Stackdriver Logging in one of the GCE VM and everything works except Log Level . Have used the parameter log_level in the file
/etc/google-fluentd/config.d/tomcat.conf
as per provided in http://docs.fluentd.org/articles/in_tail
but even then in Console Log Viewer cant able to view log in different levels. Is there any specif way to configure fluentd agent for Google Cloud?
The Google Cloud output plugin for fluentd currently uses the severity field to indicate the log level. See this issue for details. There is an open request to make it configurable.
Related
We have setup a fluentd agent on a GCP VM to push logs from syslog server (the VM) to GCP's Google Cloud Logging. The current setup is working fine and is pushing more than 300k log entries to Stackdriver (Google Cloud Logging) per hour.
Due to increased traffic, we are planning to increase the number of VMs employed behind the load balancer. However, the new VM with fluentd agent is not being able to push logs to Stackdriver. After the first time activation of VM, it does send a few entries to Stackdriver and after that, it does not work.
I tried below options to setup the fluentd agent and to resolve the issue:
Create a new VM from scratch and install fluentd logging agent using this Google Cloud documentation.
Duplicate the already working VM (with logging agent) by creating Images
Restart the VM
Reinstall the logging agent
Debugging I did:
All the configurations for google fluentd agent. Everything is correct and is also exactly similar to the currently working VM instance.
I checked the "/var/log/google-fluentd/google-fluentd.log" for any logging errors. But there are none.
Checked if the logging API is enabled. As there are already a few million logs per day, I assume we are fine on that front.
Checked the CPU and memory consumption. It is close to 0.
All the solutions I could find on Google (there are not many)
It would be great if someone can help me identify where exactly I am going wrong. I have checked configurations/setup files multiple times and they look fine.
Troubleshooting steps to resolve the issue:
Check whether you are using the latest version of the fluentd agent or not. If not, try upgrading the fluentd agent. Refer to upgrade the agent for information.
If you are running very old Compute Engine instances or Compute Engine instances created without the default credentials you must complete the Authorizing the agent procedures.
Another point to focus is, how you are Configuring an HTTP Proxy. If you are using an HTTP proxy for proxying requests to the Logging and Monitoring APIs, check whether the metadata server is reachable. The metadata server has to be reachable (and do it directly; no proxy) when Configuring an HTTP Proxy.
Check if you have any log exclusions configured which is preventing the logs from arriving. Refer Exclusion filters for information.
Try uninstalling the Fluentd agent and try to use Ops agent instead (note that syslog logs are collected by it with no setup) and check whether you were able to see the logs. Combining logging and metrics into a single agent, the Ops Agent uses Fluent Bit for logs, which supports high-throughput logging, and the OpenTelemetry Collector for metrics. Refer Ops agent for more information.
I wrote some code to automate the training procedure on our company vm instances.
you probably know that sometimes GCP can't provide you at the current moment with a machine - 'out of resource' exception.
so , I'd like to monitor which of my machines successfully turned on and which not.
if there is some way to show it on Bigquery it will be great.
thanks .
Using the Cloud Monitoring (Stackdriver) functionality is good way for monitoring all you VMs.
Here is a detailed guide to implement Monitoring on a Compute Engine Instance.
Hope you find it useful.
You can use Google cloud's activity logs too:
Activity logging is enabled by default for all Compute Engine
projects.
You can see your project's activity logs through the Logs Viewer in
the Google Cloud Console:
In the Cloud Console, go to the Logging page. Go to the Logging page
When in the Logs Viewer, select and filter your resource type from the
first drop-down list. From the All logs drop-down list, select
compute.googleapis.com/activity_log to see Compute Engine activity
logs.
Here is the Official documentation.
I am looking to split the logs on the StackDriver Agent (SDA) to multiple GCP projects (StackDrivers) based on some filter. By default SDA targets GCP project where resides.
There is a SDA configuration option, to setup different GCP destination project id, but only one.
SDA as a FluentD wrapper uses for the match section, type google_cloud.
Does this mean that the only solution is to write a custom FluentD filter that rely on the google_cloud and targets multiple GCP projects?
Thanks
First of all, you can not split logs on Stackdriver Monitoring Agent to send in different GCP project’s Stackdrivers based on any filter. I understand that you went through the document [1] and want to be confirmed about the option “type google_cloud”.
Here, the configuration options will let you override LogEntry labels [2] and MonitoredResource labels [3] when ingesting logs to Stackdriver Logging and “type google_cloud” for cloud resources of all types.
[1]:- https://cloud.google.com/logging/docs/agent/configuration#label-setup
[2]:- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
[3]:- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
If you write your own Stackdriver logger, you can do anything you want.
The Google Stackdriver logging (the driver) does not support streaming parts of logs to different Stackdriver Logs (the service).
How to disable google load balancer logging? We are currently generating 30TB (Yes TB not GB) of logs per month that we don't care about and never look at. It's quite a waste or resources.
In Google Cloud Platform console, from Stackdriver Logging section, navigate to Resource usage, choose Cloud HTTP Load Balancer and select Disable log source from the options on the right. https://console.cloud.google.com/logs/usage
You can also see option to create Exclusion filter.
Reference: https://cloud.google.com/logging/docs/exclusions#excluding-resource
To delete the previous logs
gcloud logging logs delete projects/<PROJECT-ID>/logs/requests
As per the Google Cloud Load Balancer and Stackdriver Logging documentation and also going over the Cloud Console UI and the REST API documentation, there is currently no way to disable Stackdriver logging for Google Cloud Load balancer.
Also, Google currently does not charge a fee for storing the Stackdriver logs.
If your concern is that it is hard to search through the logs, you can always filter logs while viewing on Stackdriver to help you narrow down only the items of interest.
How do I view stdout/stderr output logs for cloud ML? I've tried using gcloud beta logging read and also gcloud beta ml jobs stream-logs and nothing... all I see are the INFO level logs generated by the system i.e. "Tearing down TensorFlow".
Also in the case where I have an error that shows the docker container exited with non zero code. It links me to a GUI page that shows the same stuff as gcloud beta ml jobs stream-logs. Nothing that shows me the actual output from the console of what my job produced...
Help please??
It may be the case that the Cloud ML service account does not have permissions to write to your project's StackDriver Logs, or the Logging API is not enabled on your project.
First check whether the Stackdriver Logging API is enabled for the project by going to the API manager: https://console.cloud.google.com/apis/api/logging.googleapis.com/overview?project=[YOUR-PROJECT-ID]
Then Cloud ML service account should be automatically added as an Editor to the project, and therefore allows it to write to the project logs, but if you have changed your project permissions it may have lost it. If so, check that you've manually given the Cloud ML service account LogWriter permissions.
If you are unsure of the service account used by Cloud ML, this page has instructions on how to find it: https://cloud.google.com/ml/docs/how-tos/using-external-buckets