I was curious if Stackdriver metrics are only available via the API or is there a way to send them through Pub/Sub? I'm currently not seeing any of the metrics listed here for Compute Engine in my Pub/Sub output.
I did create a sink for all gce vm instances to export from Stackdriver logging in Pub/Sub and I'm not seeing any of them.
There are a few different types of signals that Stackdriver organizes--metrics, logs, traces, errors, plus derived signals like incidents or error groups. Logs can be exported via Pub/Sub using sinks. Metrics, traces, and errors can only be pulled via the API today.
Related
In Google Cloud Logging (nee Stackdriver), can I create a log-based metric for logs aggregated at folder/organization level? I want to have a log-based metric for a certain audit event across many projects.
This isn't currently supported. You can export logs to a different project, but can't have metrics encapsulating more than one project.
If you think that functionality should be available, you can create a Feature Request at Public Issue Tracker.
I have an on-premise app deployed in an Application Server (e.g. Tomcat) and it generates its own log file. If I decide to migrate this to an AWS EC2, including the Application Server, is it possible to port my application logs in Cloudwatch instead? or is Cloudwatch only capable of logging the runtime logs in my application server? is it a lot of work to do this or is this even possible?
Kind of confuse on Cloudwatch. Seems it can do multiple things but is it really right to make it do that? Its only supposed to log metrics right, so it can alert whatever or whoever needs to be alerted.
If you have already developed application that produces its own log files, you can use CloudWatch Logs Agent to ingest the logs into CloudWatch Logs:
After installation is complete, logs automatically flow from the instance to the log stream you create while installing the agent. The agent confirms that it has started and it stays running until you disable it.
The metrics, such as RAM usage, disk space, can also be monitored and pushed to CloudWatch through the agent.
In both cases, logs and metrics, you can setup CloudWatch Alarms to automatically detect anomalies and notify you, or perform other actions, when they are detected. For logs, this is done through metric filters:
You can search and filter the log data coming into CloudWatch Logs by creating one or more metric filters. Metric filters define the terms and patterns to look for in log data as it is sent to CloudWatch Logs. CloudWatch Logs uses these metric filters to turn log data into numerical CloudWatch metrics that you can graph or set an alarm on.
update
You can also have your application to inject logs directly to CloudWatch logs using AWS SDK. For example, in python, you can use put_log_events.
I'd like to get notifications for all standard error logs sent to Google Cloud Logging. Preferably, I'd like to get the notifications through Google Cloud Error Reporting, so I can easily get notifications on my phone through the GCP mobile app.
I've deployed applications to Google Kubernetes Engine that are writing logs to standard error, and GKE is nicely forwarding all the stderr logs to Google Cloud Logging with logName: "projects/projectName/logs/stderr"
I see the logs show up in Google Cloud Logging, but Error Reporting does not pick up on them.
I've tried troubleshooting as described here: https://cloud.google.com/error-reporting/docs/troubleshooting. But the proposed solutions revolve around formatting the logs in a certain way. What if I've deployed applications for which I can't control the log messages?
A (totally ridiculous) option could be to create a "logs-based metric" based on any log sent to stderr, then get notified whenever that metric exceeds 1.
What's the recommended way to get notified for stderr logs?
If Error Reporting is not recognizing the stderr logs from your container it means they are not displayed in the correct format for this API to be able to detect them.
Take a look at this guide on how to setup error reporting for GKE API
There are other ways to do this with third party products like gSlack where basically you will export the stackdriver logs to pub/sub and then send them into the Slack channel with Cloud Functions.
You can also try to do it by using Cloud Build and try to integrate it with GKE container logs.
Still, I think the best and easiest option will be by using the Monitoring Alert.
You can force the error by setting the #type in the context as shown in the docs. For some reason, even if this is the Google library, and it has code to detect that an Exception is thrown, with its stack trace, it won't recognize it as an error worth reporting.
I also added the service array to be able to identify my service in the error reporting.
I need to fetch the GCP metrics without using stackdriver api(without enabling the stackdriver api). Ex. Pub/Sub metrics -> pubsub.googleapis.com/subscription/ack_message_count.
You can interact directly with the Stackdriver Monitoring API to retrieve the metrics you're interested in.
However, this requires the Stackdriver Monitoring API to be enabled since this will start the metric collection (which is needed before you can start fetching them).
Having the API enabled has the additional advantage that has support for the Client Libraries, making metrics fetching easier.
As a partial workaround and specific for ack_message_count, you can use the libraries to take advantage of the seek method available in the PubSub API to get the retained messages in any given subscription:
Messages retained in the subscription that were published before this time are marked as acknowledged, and messages retained in the subscription that were published after this time are marked as unacknowledged. Note that this operation affects only those messages retained in the subscription
This implies that the data is constantly pulled and stored (if needed) in your own persistence layer for latter usage/aggregation.
What is the best way to set up Dataflow resource monitoring and alerting on Dataflow errors?
Is it custom log based metrics only?
Checked Cloud Monitoring - Dataflow is not listed there - no metrics available.
Checked Error Reporting - it is empty too, despite a few of my flows failing.
What do I miss?
Update March 2017: Stackdriver Monitoring Integration with Dataflow is now in Beta. Review user docs, and listen to Dataflow engineers talking about it at GCP Next.
For the time being you could set up alerts based on Dataflow logs (go to Stackdriver Logging and set up alerts there). We are also working on better alerting using Stackdriver Monitoring and will post an announcement to our Big Data blog when it's in beta.