I went through the docs but I couldn't find a way of sending an alert based on my Cloud function logs, is this possible?
What I want to do is trigger an alert when I get an specific log node.
Like, every time I have this:
Triggers an alert in my GCP, is that possible?
Create an log-based metric from the advanced filter of the Logs and create an alerting from the log-based metric. To create alert from the log-based metric you need to click on three dot right side of the metric name there you will see the create alert option.
As an alternative to log-based metrics, now log-based alerts are also supported. This lets you receive notification whenever a specific message appears in logs.
Related
We host quite a few things on our GCP project and it's kinda nice to be alerted on new errors, but I wish to send email notifications to Pagerduty only from my production kubernetes cluster.
Is there a way to do this, or should I filter this somehow in pagerduty (unsure if possible - still new with it).
Here is the procedure of sending notifications from kubernetes to pagerduty:
A metric needs to be created based on the requirement and that metric needs to be added when we create an alert. When we proceed further in the notifications page you can select pagerduty and proceed further in creating alerts.
Step1:
Creating an log-based-Metric :
1.In console go to log-based metrics page and there go to create metric and create a new custom metric.
2.Here set the metric type as counter and in details add log metric name as (user/delete).
3.In metric give the query which can fetch the logs of the errors you are expecting to be alerted and create the metric.
Step 2:
Creating an alert policy :
1.In the console go to the Alerting page and there go to create policy and create a new alerting policy.
2.Go to add condition and in that resource type is the resource which we need to be triggered (in our case kubernetes pod) and metric is the metric we created in step1.
3.In filer add the project id and in period add the suitable period. Next in the configuration add these details and proceed to the next steps leaving the rest fields as default.
Step3:
1.Next you will be directed to select notification channels there you go to Manage notification channels and select pager duty services and add new and there add the display name and the service key and check connectivity save and proceed further.
2.Add the alert name and save the alert.
Is it possible to create alerts for configuration activities?
On the dashboard of my GCP project, I'm able to see the history of activities. However, for security reasons, I would like to be able to receive notifications when certain activities happen, e.g. Set IAM policy on project, deleting instance of project, etc. Is this possible?
I have looked into "metric-based alerting policies", but I'm only able to create alerts for uptime checks. Not sure what else to look for.
You are on the right path. You need to create a log-based metric and then to create an alert when the counter cross a threshold (1 for example)
Now a more straightforward solution is available: In one step, you can use log-based alerts. It allows to set alerts on any log type and content. This new feature is on preview and was announced few days ago.
We have configured tracking in cloudwatch for sending emails via AWS SES programmatically via AWS SDK ( as mentioned in the documentation). Whenever I send an email for the first time with a new configuration set I am able to see the metrics (open, click) value updated to 1. When I send the same email to another recipient the metric value should increase to 2 but there is no change in the cloudwatch metric and it always shows value as 1.
I have configured message tags and configuration set as well.
I checked after few hours but still, metrics were not updated. I am not sure if there is some issue with cloudwatch - SES configuration or I am missing any configuration in the graph?
You can check in the cloudwatch graphed metrics. You may have selected Statistic value to Average which is default selection. You can try after changing that value to Sum. View Example here
I can find a graph of "Group size" in the page of the instance group.
However, when I try to find this metric in Stackdriver, it doesn't exist.
I tried looking in the metricDescriptors API, but it doesn't seem to be there either.
Where can I find this metric?
I'm particularly interested in sending alerts when this metrics goes to 0.
There is not a Stackdriver Monitoring metric for this data yet. You can fetch the size using the instanceGroups.get API call. You could create a system that polls this data and posts it back to Stackdriver Monitoring as a custom metric and then you will be able to access it from Stackdriver.
Having created log-based metrics in cloud console, I then want to create alerts so that every time there is a new matching log entry, the alert triggers.
In trying to create a suitable metric, the most likely looking options seem to be threshold or rate of change, but I don't think either will work for a policy of 1 log message => 1 alert.
Help appreciated.
Yes, today the only way to alert on the log message is to create a threshold condition on the log metric with a very small threshold (0.001) and a duration of 1 minute.
Thanks for using Stackdriver.
You can use another alert triggering software (like PagerDuty) which is pinged by emails sent by Stackdriver. PagerDuty is able to filter all those emails that has the word RESOLVE in its subject. They can be absolutely thrown away in our case if you'd like to avoid autoresolving. Of course Stackdriver and PagerDuty alerts will diverge from each other (states will be inconsistent) but you should consider PD as single source of truth this time. It could be a possible workaround.
With log-based alerts, we can create alerts from the logs, incident will be creatied for each matching entry.
https://cloud.google.com/blog/products/operations/create-logs-alerts-preview