We are trying to use the Stackdriver Agent to send metrics from an application that is running in a container. We manage to run the agent as a sidecar container and it is pushing metrics successfully.
The problem is it is pushing them under an instance_id that doesn't make sense for us. What we want to do is simply configure the instance_id on our own.
My guess was we could do this by configuring the Hostname parameter in collectd-gcm.conf.tmpl but it seems like the write_gcm plugin is ignoring that. Next thing we could try is to configure the write_gcm plugin ourselves but I can't find any documentation or source code on that plugin to see how it works and what configuration options does it support. Any ideas where to go from here?
Our configuration looks like this now:
Interval 60
Hostname "ingestion"
LoadPlugin write_gcm
<Plugin "write_gcm">
PrettyPrintJSON false
</Plugin>
Include "/opt/stackdriver/collectd/etc/collectd.d"
Instance-ids are default metadata entries. As per documentation, "Default metadata is always defined and set by the server. You cannot manually edit any of these metadata pairs." Currently, it is not possible to edit the instance-id or customize it. You can submit a feature-request on Google Cloud public issue tracker along with your use case, requirement and expected outcome.
Related
I Google Cloud's ErrorReporting the "Seen in" section doesn't show anything useful for my GKE deployments. It's either empty or says gke_instance which is pretty useless. I have set the serviceContext correctly in my logs and the container name is also set in the labels of the log entries and yet it's not showing up. Is this a bug or am I missing something obvious here?
To resolve your issue try below 3 solutions :
Solution 1 : If you're using Legacy Stackdriver disable it and enable Stackdriver Kubernetes Engine Monitoring, for more information refer to the similar Stack Question.
Solution 2 : As stated by this release notes, the stackdriver agent actually becomes disabled by default in 1.15. To activate it again you need to edit the cluster following these instructions. Also refer to this Stack Question.
Solution 3 : If a new GKE cluster has Cloud Operations for GKE set to System and workload logging and monitoring, however no application logs are showing up refer to this Stack Question.
Note : Issue was the node pool using the default service account (which no longer existed). Created a new node pool following the document.
How have people implemented Production Access Controls (i.e. logging and reporting on access to compute instances by services and humans over SSH). Our goal is to forward all user logon entries to our SIEM consistently across projects and ideally avoid having project specific Stackdriver sinks (and associated setup and maintenance).
We've tried the following:
Enabled auth log forwarding in Fluentd as only syslog is done by default
Enabled organization level sinks that send to a topic (to forward on to SIEM via HTTP subscriber) that include all children
Can see syslog/auth at the project level for non-Container OS images (i.e. Ubuntu)
Issues we're seeing:
- Limited documentation on filter format at org level (seems to differ from project level for things like logName). log_id function does appear to work
- Some log types appear at the org level (things like cloudapis activity) but syslog does not appear to get processed
- Container OS appears to not enable ssh/sudo forwarding by default in fluentd (or I haven't found which log type has this data). I do see this logged to journalctl on a test node
Does anyone have a consistent way to achieve this?
In case anyone else comes across this, we found the following:
It is possible to set up Stackdriver sinks at org level through CLI. Not visible through Cloud Console UI and also CLI does not allow you to list log types at org
Filters can be defined on the sinks in addition to logName but format can differ to project level filters
You need to enable auth log logging in fluentd which is platform specific (i.e. one process for google-fluentd on Ubuntu is different to stackdriver setup on Container OS)
SSHD for some reason does not log the initial log stating user and IP through syslog (and thus fluentd) and therefore is not visible to Stackdriver
Use or org sinks to topics is a child project with subscription to forward to your SIEM of choice, works well
Still trying to get logs of gcloud ssh commands
A way to approach this could be to by exporting your log sink to BigQuery. Note that sink setup to export BigQuery Logs for all projects under the Organization contains a parameter that is set to 'False', the field 'includeChildren' must be set to 'True'. Once set to true then logs from all the projects, folders, and billing accounts contained in the sink's parent resource are also available for export, if set to false then only the logs owned by the sink's parent resource are available for export. Then you must be able to filter the logs needed from BigQuery.
Another way to approach this will be to script it out by listing all the projects using command: gcloud projects list | tail -n +2 | awk -F" " '{print $1}' This can be made into an array that can be iterated over and the logs for each project can be retrieved using a similar command as the one in this doc.
Not sure if all this can help somehow to solve or workaround your question, hope so.
We have multiple spring boot and python apps running on top of GKE and for spring boot applications am using spring-cloud-gcp-starter-trace to log traces to stack driver so that I can debug those traces via the stack driver UI.
Am not able to figure out how to add labels like service_name, service_version and cluster_name so that I can filter out only those traces for reporting purposes because right now we have istio configured on one cluster and even with one percent sampling rate it's generating tons of telemetry data and with UN-availability of filters or am missing some configuration, the trace UI has almost become useless for me
I had a look at the documentation for spring-cloud-gcp-starter-trace, they don't have any properties through which I can set these fields, Am setting app name and app version via the metadata tags of the kubernetes deployment template but they aren't getting picked up.
Can some one please let me know how can I achieve this.
You can add custom tags using the brave.SpanCustomizer. Just autowire it in as the bean already exists in the application context.
You can then add tags like this:
#Autowired
SpanCustomizer spanCustomizer;
...
spanCustomizer.tag("my-tag", "my tag value");
These will turn into labels on you traces in Stackdriver Trace, on which you can search.
If you're using OpenCensus, you can use annotations to pass metadata into the Trace backend:
https://cloud.google.com/trace/docs/setup/java#custom_spans.
I don't see anything in spring-cloud-gcp-starter-trace documentation (what little I could find) regarding annotations however.
Want to find logs for Metadata added for google cloud project i.e. Project Metadata, not for compute/VM instance metadata.
Tried to find in stack-driver logging but it has showing only for compute instances like as compute.instances.setMetadata or compute.instances.insert or compute.instances.delete etc.
I am looking for metadata or property added in/for GCP project(Not for VM instance metadata). Reason behind this, someone is adding/modifying property and we are unable to find the history to track this change and it causes to failure of application.
For future readers, you can add the following to your query to look for project metadata:
protoPayload.methodName="v1.compute.projects.setCommonInstanceMetadata"
You could try looking at the Activity page - https://console.cloud.google.com/home/activity
The Logs console also has a Google Project resource that you can filer on.
How to change the log level of Java profiler? I am running the profiler outside GCP.
Although, Profiler is working fine. It is repeatedly logging following errors:
E0803 12:37:37.677731 22 cloud_env.cc:61] Request to the GCE metadata server failed, status code: 404
E0803 12:37:37.677788 22 cloud_env.cc:148] Failed to read the zone name
How can I disable these logs?
For Stackdriver Logging you can use log exclusion filters to create customised filters for logs you want to exclude.
In the Logs Viewer panel, you can enter a filter expression that matches the log entry you want to exclude. This documentation explains about various interfaces to create filters.
You may also want to export the log entries before excluding them, if you do not want to permanently lose the excluded logs.
With respect to this issue in general (i.e. for third party logging), I went ahead and created a feature request on your behalf. Please star it so that you could receive updates about this feature request and do not hesitate to add additional comments to provide details of the desired implementation. You can track the feature request by following this link.