How to change the log level of Java profiler? I am running the profiler outside GCP.
Although, Profiler is working fine. It is repeatedly logging following errors:
E0803 12:37:37.677731 22 cloud_env.cc:61] Request to the GCE metadata server failed, status code: 404
E0803 12:37:37.677788 22 cloud_env.cc:148] Failed to read the zone name
How can I disable these logs?
For Stackdriver Logging you can use log exclusion filters to create customised filters for logs you want to exclude.
In the Logs Viewer panel, you can enter a filter expression that matches the log entry you want to exclude. This documentation explains about various interfaces to create filters.
You may also want to export the log entries before excluding them, if you do not want to permanently lose the excluded logs.
With respect to this issue in general (i.e. for third party logging), I went ahead and created a feature request on your behalf. Please star it so that you could receive updates about this feature request and do not hesitate to add additional comments to provide details of the desired implementation. You can track the feature request by following this link.
Related
How have people implemented Production Access Controls (i.e. logging and reporting on access to compute instances by services and humans over SSH). Our goal is to forward all user logon entries to our SIEM consistently across projects and ideally avoid having project specific Stackdriver sinks (and associated setup and maintenance).
We've tried the following:
Enabled auth log forwarding in Fluentd as only syslog is done by default
Enabled organization level sinks that send to a topic (to forward on to SIEM via HTTP subscriber) that include all children
Can see syslog/auth at the project level for non-Container OS images (i.e. Ubuntu)
Issues we're seeing:
- Limited documentation on filter format at org level (seems to differ from project level for things like logName). log_id function does appear to work
- Some log types appear at the org level (things like cloudapis activity) but syslog does not appear to get processed
- Container OS appears to not enable ssh/sudo forwarding by default in fluentd (or I haven't found which log type has this data). I do see this logged to journalctl on a test node
Does anyone have a consistent way to achieve this?
In case anyone else comes across this, we found the following:
It is possible to set up Stackdriver sinks at org level through CLI. Not visible through Cloud Console UI and also CLI does not allow you to list log types at org
Filters can be defined on the sinks in addition to logName but format can differ to project level filters
You need to enable auth log logging in fluentd which is platform specific (i.e. one process for google-fluentd on Ubuntu is different to stackdriver setup on Container OS)
SSHD for some reason does not log the initial log stating user and IP through syslog (and thus fluentd) and therefore is not visible to Stackdriver
Use or org sinks to topics is a child project with subscription to forward to your SIEM of choice, works well
Still trying to get logs of gcloud ssh commands
A way to approach this could be to by exporting your log sink to BigQuery. Note that sink setup to export BigQuery Logs for all projects under the Organization contains a parameter that is set to 'False', the field 'includeChildren' must be set to 'True'. Once set to true then logs from all the projects, folders, and billing accounts contained in the sink's parent resource are also available for export, if set to false then only the logs owned by the sink's parent resource are available for export. Then you must be able to filter the logs needed from BigQuery.
Another way to approach this will be to script it out by listing all the projects using command: gcloud projects list | tail -n +2 | awk -F" " '{print $1}' This can be made into an array that can be iterated over and the logs for each project can be retrieved using a similar command as the one in this doc.
Not sure if all this can help somehow to solve or workaround your question, hope so.
I am having difficulties integrating with Stackdriver Error Reporting.
When using the stack driver log viewer I can see that it has correctly identified it as an error event due to the orange '!!' on the log line.
The logs are coming from a Java Application in a pod on Kubernetes. I am using SLF4J and logback to control my logging. I realise this is not the example in the docs which suggest fluentd however I would like to avoid changing my logging across all applications.
Following the troubleshooting guide I am able to submit a log line that is picked up and also directly report an error. This makes me think the issue must be permissions related. I have tried adding the "Error Reporting Admin" role onto the compute engine default service account and onto Kubernetes Engine Service Agent but this has not worked.
Am I missing something?
The !! in the logs viewer means that the LogEntry.severity field has a value of ERROR (which is provided by the client that wrote the entry). Entries that land in Error Reporting need to meet a few other criteria: https://cloud.google.com/error-reporting/docs/formatting-error-messages
You might also be interested in the details on how errors are grouped together: https://cloud.google.com/error-reporting/docs/grouping-errors
The "Error Reporting Admin" role would allow someone (or a service account) to perform actions like muting an error group. There are no permissions requirements to get data from logging into Error Reporting.
I am new to Google Cloud platform, I am trying out Google's Talent Solution APIs and getting 400 and 409 errors. I would like to know how to see the error trace on my google cloud platform console for those requests? I could see those requests in the dashboard, not sure if I can drill down and see the exact error?
If you see these errors in your Developer Console main dashboard in the "Error Reporting" section, you should be able to click on the "Go to Error Reporting" link. There, you'll see a list of errors that happened lately. Clicking on one of them, you'll see more info on that specific error (occurrences, last seen, etc...) and a stack trace sample. Further down, you'll see a list of the last occurrences for that error and their respective link to the logs.
Alternatively, you may go directly to your Dev Console -> Logging section and choose the Talent Solution resource if available or the resource where you call the API from (App Engine, GKE, etc...) and filter the logs by error code to get details about these.
We are trying to use the Stackdriver Agent to send metrics from an application that is running in a container. We manage to run the agent as a sidecar container and it is pushing metrics successfully.
The problem is it is pushing them under an instance_id that doesn't make sense for us. What we want to do is simply configure the instance_id on our own.
My guess was we could do this by configuring the Hostname parameter in collectd-gcm.conf.tmpl but it seems like the write_gcm plugin is ignoring that. Next thing we could try is to configure the write_gcm plugin ourselves but I can't find any documentation or source code on that plugin to see how it works and what configuration options does it support. Any ideas where to go from here?
Our configuration looks like this now:
Interval 60
Hostname "ingestion"
LoadPlugin write_gcm
<Plugin "write_gcm">
PrettyPrintJSON false
</Plugin>
Include "/opt/stackdriver/collectd/etc/collectd.d"
Instance-ids are default metadata entries. As per documentation, "Default metadata is always defined and set by the server. You cannot manually edit any of these metadata pairs." Currently, it is not possible to edit the instance-id or customize it. You can submit a feature-request on Google Cloud public issue tracker along with your use case, requirement and expected outcome.
We are starting to enable Stackdriver for our project, but while I have a ubuntu instance with stackdriver logging, and I am getting the logs shipped back (I can see the logs in Stackdriver Logging), when I browse to Error Reporting, it's just a blank screen with a button to "Setup Error Reporting", which takes me to some API documentation which I think is tailored for new application coding. We are running nginx and the logging is working, but I can't for the life of me figure out how to get the Error Reporting to work properly, if that's even doable.
"Setup Error Reporting" should guide you to the setup documentation (not API documentation). Depending on the platform you are using, you might need to perform some changes in your application's code or log format. Read more at https://cloud.google.com/error-reporting/docs/setup/
If you have Stackdriver Logging setup and on Google Compute Engine, the requirement is for your exception stack traces to be log in single log entries.