I am looking for ideas on how to set the recipient of PagerDuty alerts.
To give some context, I have an aws config rule that publishes a new event into an SNS topic, via EventBridge, each time the config rule is non-compliant then I have PagerDuty subscribed to the sns topic; PagerDuty successfully receives the alerts and forward them to the alert recipients, no issue is here.
My question is this: is it possible to set the recipient of the PagerDuty Alert based on the event that triggers the alert?
I am thinking about using lambda to query CloudTrail to extract the email address of the user initiating the event that causes the aws config to become non-compliant, but not sure how to set that email address as the recipient of the PagerDuty notification.
Is this even possible? or is there a better way to approach it?
Thanks in advance
Some options for thought:
Depending on the size of your instance you could build a specific service for each of the possible recipients. Either using the lambda you mentioned to control which service the alert is routed to. Or, alternatively, using a PagerDuty global Ruleset (or event orchestration) to route the alert based on its contents.
This doesn't need a much setup initially but the tradeoff is that it quickly becomes unwieldly at scale.
https://support.pagerduty.com/docs/event-orchestration#global-orchestrations
I've also seen solutions that assign an escalation policy without a specific target to a service such as user account with no contact info. When an alert and incident are opened a webhook is sent to, for example, RunDeck and that tool takes action in PagerDuty. The correct recipient is assigned to the incident and requested to acknowledge.
The tradeoffs here are losing visual sight of who is on-call for a service and the lift to stand up RunDeck, a lambda, or some other listener to process the webhook event.
https://support.pagerduty.com/docs/event-orchestration#webhooks
https://www.pagerduty.com/integrations/rundeck-runbook-automation/
Related
2 weeks back, I saw a notification in AWS saying, the one of the 3 AWS EC2 instance will be stopped because of the underlying hardware cause. I almost missed the dead line as I didn't the notification via email. Is there a way get these notification via email?
It would be much better if I can get the notifications using cloud watch to my email.
If you're on Business or Enterprise Support, AWS provides a solution named "AWS Health Aware" built on top of AWS Health API, it does exactly what you're after sends you notifications via several methods including emails regarding outages and planning maintenance that affect your account(s).
If you're on a Basic support plan you can easily built something similar yourself using based on [AWS Health events][2]
This is an example of capturing AWS Health events via EventBridge so they can be further send to a lambda or SNS
https://docs.aws.amazon.com/health/latest/ug/cloudwatch-events-health.html
What I've done in the is following:
Capturing AWS Health events via EventBridge in spoke accounts and forward them to the event bus in my hub account
In the hub account I would subscribe a lambda function for all events with source "aws.health" and then send email or slack message whatever works for you.
I think Amazon EventBridge is what you are looking for. With EventBridge you can detect and react to different events such as EC2 instances scheduled for updates. You can, for example, create a rule for detecting a specific health event and in the target section choose to receive an email through SNS or invoke a CloudWatch log group, etc. Read more here.
I am setting up alerting for GCP VMs. It works fine for email, but I'm trying to use the webhook option. It shows the incident was caught and the webhook triggered. But I don't see an alert on the receiving end. I don't know how to debug since GCP webhooks seems like a blackbox. Does anybody know where I can see the log for the actual webhook call? I'm not sure it is receiving an alert ID from the webhook call.
I'm using this document:
https://cloud.google.com/monitoring/alerts/using-channels-api#api-create-channels
Thanks!
Gary
You configure a webhook notification channel and expect to be notified when incidents occur, but you might not receive any notifications because of following reasons:
1.Private endpoint
You can't use webhooks for notifications unless the endpoint is public.
To resolve this situation, use Pub/Sub notifications combined with a pull subscription to that notification topic.
When you configure a Pub/Sub notification channel, incident notifications are sent to a Pub/Sub queue that has Identity and Access Management controls. Any service that can query for, or listen to, a Pub/Sub topic can consume these notifications. For example, applications running on App Engine, Cloud Run, or Compute Engine virtual machines can consume these notifications.
2.Public endpoint
To identify why the delivery failed, examine your Cloud Logging log entries for failure information.
For example, you can search for log entries for the notification channel resource by using the Logs Explorer, with a filter like the following:
resource.type="stackdriver_notification_channel"
NOTE : Also check whether you have been mentioned in the recipient list, if not mentioned you will not be able to see the alert on the receiving end.
Refer Troubleshooting Alerting policies for information.
Im using AWS Lambda for sending emails by Amazon SES.
Each time that the request send_email succeeds I got the message id like confirmation.
But now, I want to use somehow this messageId. For example to know if this message concretelly get delivered or bounced.
CloudWatch offers me metrics, with the number of delivered or bounced messages but I want to look throught each one.
¿Is there any statement that allows to know the state of a message by id?
Due to the asynchronous nature of SMTP and how mails are being routed, you can't have this information immediately when sending an e-mail (via SES or otherwise).
However, AWS provides a feedback system which notifies an SNS topic when a message is delivered, bounced, or a complaint was received.
While the documentation is very detailed, the general idea is that you subscribe your own lambda(s) to the configured SNS topic(s) and you can then handle these events as you see fit (i.e. persist data on DynamoDB, call your unsubscribe API and so on).
I know that the AWS sends emails about any EC2 instances with Scheduled Events, but I would also like to have these notifications sent through HipChat and text message as well. Is there a default way of selecting a setting that can do this in AWS? Or would I have to create a more custom solution to achieve this?
With Scheduled Events in EC2 being manual tool for monitoring health of instances as of right now, there isn't many options. From the design point of view on AWS platform, this could be one possible solution:
Create AWS SES mailbox and enable it to receive emails,
Create a forwarding rule in your original e-mail inbox where you receive those e-mail notifications from AWS about Scheduled Events to forward e-mails of this type (based on filter you define) to the previously created AWS SES mailbox,
In your AWS SES mailbox create "Receipt Rule" with SNS Action, that will basically publish entire content of the e-mail to the SNS Topic using AWS SNS. (The SNS topic you choose must be in the same AWS region as the Amazon SES endpoint you use to receive email)
Once that e-mail content lands in AWS SNS Topic as an event, you can use AWS Lambda subscribed to your SNS topic as a handler of the event, parse what you need and forward customized notification to any of your third-party (outside of AWS) alerting tools (such as PagerDuty, Slack channel, HipChat, or whatever you use for SMS notifications).
This design does have one week point: relying onto your external mailbox forwarding system (if you are receiving Scheduled Events e-mails in inbox not based on AWS SES).
Ideally you could try to see in your AWS Account settings if it is possible to use AWS SES mailbox for Scheduled Events notifications, but I haven't tried that.
You can receive text messages as notification by setting alarm for that ec2 instance e.g if CPU usages goes above 80% then create an alarm and eventually you can set it to get emails or text messages. Thanks
We're using AWS SNS (Simple Notification Services) as one of the mechanisms of sending out notifications for users. That means, there are notifications sent through SMS (using AWS SNS) / Emails / automated calls.
But any time, if a user unsubscribes from any of these three methods, We should stop sending further alerts to the user in all the three modes.
Currently I don't have a way in API to check if a subscription request has been opted-out (by sending a STOP message), so that we can block the other two modes of communications as well.
Is it possible, in SNS? I've tried looking into the list of API, console and Amazon Forums.
I've figured out that this below behaviour of AWS SNS (for SMS end points) can be a workaround.
Although currently there is no way of getting a list of endpoints (users) who unsubscribe by responding STOP to 30304 (Amazon's short number.)
But, when we subscribe a user for a topic, AWS creates a subscription record with subscriptionArn = "PendingConfirmation"
If the user responds STOP (opts out), the subscription record is deleted from the topic. If the user subscribes to it, the subscription's subscriptionArn changes to some valid ARN (Amazon's resource Number) from "PendingConfirmation".
ListSubscriptionsByTopic API lists the user subscription even if it is in "PendingConfirmation" state.
So for now, only way to know if a user has unsubscribed from the topic would be, maintaining a list of users with their end points (phone numbers / email ids) and comparing it at later point of time with the list of available list of subscriptions. If something has disappeared, it means they've unsubscribed.
Of course, it can't be real-time. But this is the only workaround that is available for now.
As an additional info, if a subscription is pending for more than 72 hours, it is automatically deleted.