How have people implemented Production Access Controls (i.e. logging and reporting on access to compute instances by services and humans over SSH). Our goal is to forward all user logon entries to our SIEM consistently across projects and ideally avoid having project specific Stackdriver sinks (and associated setup and maintenance).
We've tried the following:
Enabled auth log forwarding in Fluentd as only syslog is done by default
Enabled organization level sinks that send to a topic (to forward on to SIEM via HTTP subscriber) that include all children
Can see syslog/auth at the project level for non-Container OS images (i.e. Ubuntu)
Issues we're seeing:
- Limited documentation on filter format at org level (seems to differ from project level for things like logName). log_id function does appear to work
- Some log types appear at the org level (things like cloudapis activity) but syslog does not appear to get processed
- Container OS appears to not enable ssh/sudo forwarding by default in fluentd (or I haven't found which log type has this data). I do see this logged to journalctl on a test node
Does anyone have a consistent way to achieve this?
In case anyone else comes across this, we found the following:
It is possible to set up Stackdriver sinks at org level through CLI. Not visible through Cloud Console UI and also CLI does not allow you to list log types at org
Filters can be defined on the sinks in addition to logName but format can differ to project level filters
You need to enable auth log logging in fluentd which is platform specific (i.e. one process for google-fluentd on Ubuntu is different to stackdriver setup on Container OS)
SSHD for some reason does not log the initial log stating user and IP through syslog (and thus fluentd) and therefore is not visible to Stackdriver
Use or org sinks to topics is a child project with subscription to forward to your SIEM of choice, works well
Still trying to get logs of gcloud ssh commands
A way to approach this could be to by exporting your log sink to BigQuery. Note that sink setup to export BigQuery Logs for all projects under the Organization contains a parameter that is set to 'False', the field 'includeChildren' must be set to 'True'. Once set to true then logs from all the projects, folders, and billing accounts contained in the sink's parent resource are also available for export, if set to false then only the logs owned by the sink's parent resource are available for export. Then you must be able to filter the logs needed from BigQuery.
Another way to approach this will be to script it out by listing all the projects using command: gcloud projects list | tail -n +2 | awk -F" " '{print $1}' This can be made into an array that can be iterated over and the logs for each project can be retrieved using a similar command as the one in this doc.
Not sure if all this can help somehow to solve or workaround your question, hope so.
Related
I am using #google-cloud/logging package to get logs from gcloud, and it works nicely, you can get logs, event (and query them if needed). But how I can get the same info as Logs Explorer? I mean different type of fields which can be queried and etc:
On this picture you see Log fields like, FUNCTION NAME which may be a list of values. And it seems that #google-cloud/logging can't get this meta (or fields info)? So is it possible to obtain it using some other APIs?
If I understand your question correctly, you're asking how Logs Viewer is determining the values that allows it to present you with the various log fields to filter|refine your log queries.
I suspect (don't know) that the viewer is building these lists from the properties as it parses the logs. This would suggest that, the lists are imperfect and that e.g. FUNCTION_NAME's would only appear once a log including the Function's name were parsed.
There is a way to enumerate definitive lists of GCP resources. This is done using list or equivalent methods available using service-specific libraries (SDKs) e.g. #google-cloud/functions.
The easiest way to understand what functionality is provided by a given Google service is to browse the service using Google's APIs Explorer. Here's Cloud Logging API v2 and here's Cloud Functions API.
You can prove to yourself that there's no method under Cloud Logging that allows enumeration of all a project's Cloud Functions. But there is a method in Cloud Functions projects.locations.functions.list. The latter returns a response body that includes a list of functions that are a type CloudFunction that have a name.
Another way to understand how these APIs ("libraries") are used is to add --log-http to any gcloud command to see what API calls are being made by the command.
What is the most efficient way to update all assets labels per project?
I can list all project resources and their labels with gcloud asset search-all-resources --project=SomeProject. The command also returns the labels for those assets.
Is there something like gcloud asset update-labels?
I'm unfamiliar with the service but, APIs Explorer (Google's definitive service documentation), shows a single list method.
I suspect (!?) that you will need to iterate over all your resource types and update instances of them using any (there may not be) update (PATCH) method that permits label changes for that resource type.
This seems like a reasonable request and you may wish to submit a feature request using Google's issue tracker
gcloud does not seem to have a update-labels command.
You could try the Cloud Resource Manager API. For example, call the REST or Python API: https://cloud.google.com/resource-manager/docs/creating-managing-labels#update-labels
Help definition is not clear for me:
Token used to route traces of service requests for investigation of
issues.
Could you provide simple example how to use it?
I tried:
gcloud compute instances create vm3 --trace-token xyz123
I can find "vm3" string in logs, but not my token xyz123.
The only use of it seems to be in grep:
history| grep xyz123
The flag --trace-token is intended to be used by the support agents when there is some error which is difficult to track from the logs. The Google Cloud Platform Support agent provides a time bound token which will expire after a specified time and asks the user to run the command for the specific product in which the user is facing the issue. Then it gets easier for the support agent to trace the error by using that --trace-token.
For example :
A user faced some error while creating a Compute Engine instance and contacted the Google Cloud Platform Support team. The support agent then inspected the logs and other resources but could not find the root cause of the issue. Then the support agent provides a --trace-token and asks the user to run the below command with the provided --trace-token.
--trace-token = abcdefgh
Command : gcloud compute instances create my-vm --trace-token abcdefgh
After the user runs the above command the support agent could find the error by analysing in depth with the help of the --trace-token
Please note that when a --trace-token flag is used the content of the trace may include sensitive information like auth tokens, the contents of any accessed files. Hence they should only be used for manual testing and should not be used in production environments.
On Windows 10 I'm using the AWS S3 Java SDK v2, Maven coordinates software.amazon.awssdk:s3:2.7.5 (from software.amazon.awssdk:bom). I have my credentials stored in ~user/.aws/credentials.
As per the docs, the I use an S3Client.putObject() to put an object in a bucket. I get two DEBUG level log messages from software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain:
software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
However S3Client.putObject() succeeds.
Don't I have my credentials configured properly by putting them in ~user/.aws/credentials? Why is this error being logged? I realize that it is a DEBUG level message, but it clutters up the log, not only once but twice. Plus there was no problem; the credentials were configured correctly.
Is there some way to turn off this log message? Or is there some extra configuration step I'm not doing correctly?
Update: It appears that this debug message is only generated once for the creation of the S3 client, so it's not as intrusive as it appears. Still it's more cluttered than desired to log an exception for only a warning condition, even though it's only at a DEBUG level.
This is a common error message when you’re using ENV vars to provide your service with an AMI account (accessid-accesskey combination) as you’ve mentioned is DEBUG. I don’t usually use this log level on environments except for development.
Anyway you can configure different log levels for different namespaces, Generally I tend to use “Information” on third party libraries (except for Entity framework that always gets set to Error because of its verbosity).
How to change the log level of Java profiler? I am running the profiler outside GCP.
Although, Profiler is working fine. It is repeatedly logging following errors:
E0803 12:37:37.677731 22 cloud_env.cc:61] Request to the GCE metadata server failed, status code: 404
E0803 12:37:37.677788 22 cloud_env.cc:148] Failed to read the zone name
How can I disable these logs?
For Stackdriver Logging you can use log exclusion filters to create customised filters for logs you want to exclude.
In the Logs Viewer panel, you can enter a filter expression that matches the log entry you want to exclude. This documentation explains about various interfaces to create filters.
You may also want to export the log entries before excluding them, if you do not want to permanently lose the excluded logs.
With respect to this issue in general (i.e. for third party logging), I went ahead and created a feature request on your behalf. Please star it so that you could receive updates about this feature request and do not hesitate to add additional comments to provide details of the desired implementation. You can track the feature request by following this link.