I am trying to edit an AWS CloudWatch alert that sends an email to my team such that custom content is sent in the email. Currently, all email alerts contain only auto-generated content. The email content contains the reason for the alert, a link to the alarm in the AWS console, and sections for Alarm Details, Threshold, and Monitored Metric. However, I want to add custom content listing likely causes of the alert and procedures to execute when receiving the alert. Does anybody know how custom content can be added to a CloudWatch alert email?
I have read existing AWS CloudWatch Alarm documentation such as How to Create/Edit a CloudWatch Alarm, and How to create a CPU Usage Alarm that Sends an Email. I have also tried various Google searches and searches for existing questions here on SO but to no avail. Any help/advice would be greatly appreciated.
You can setup a Lambda trigger to the alarm and send an email using AWS SES SMTP credentials, creating a formatted email content with alarm trigger event data.
You can flow the document you mentioned above: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_AlarmAtThresholdEC2.html
Choose existing SNS topic or create new one, after that enter slack email and when have any alarms will be sent to Slack channel that was integrated Email app by you
xxxxxxxxxx#x.slack.com you can use add Email App in Slack like icon below
Related
I am looking for ideas on how to set the recipient of PagerDuty alerts.
To give some context, I have an aws config rule that publishes a new event into an SNS topic, via EventBridge, each time the config rule is non-compliant then I have PagerDuty subscribed to the sns topic; PagerDuty successfully receives the alerts and forward them to the alert recipients, no issue is here.
My question is this: is it possible to set the recipient of the PagerDuty Alert based on the event that triggers the alert?
I am thinking about using lambda to query CloudTrail to extract the email address of the user initiating the event that causes the aws config to become non-compliant, but not sure how to set that email address as the recipient of the PagerDuty notification.
Is this even possible? or is there a better way to approach it?
Thanks in advance
Some options for thought:
Depending on the size of your instance you could build a specific service for each of the possible recipients. Either using the lambda you mentioned to control which service the alert is routed to. Or, alternatively, using a PagerDuty global Ruleset (or event orchestration) to route the alert based on its contents.
This doesn't need a much setup initially but the tradeoff is that it quickly becomes unwieldly at scale.
https://support.pagerduty.com/docs/event-orchestration#global-orchestrations
I've also seen solutions that assign an escalation policy without a specific target to a service such as user account with no contact info. When an alert and incident are opened a webhook is sent to, for example, RunDeck and that tool takes action in PagerDuty. The correct recipient is assigned to the incident and requested to acknowledge.
The tradeoffs here are losing visual sight of who is on-call for a service and the lift to stand up RunDeck, a lambda, or some other listener to process the webhook event.
https://support.pagerduty.com/docs/event-orchestration#webhooks
https://www.pagerduty.com/integrations/rundeck-runbook-automation/
I am new to alert policy creation in google cloud.
I have set up a GKE cluster and enabled upgrade notifications to publish a message to Pub/Sub topic whenever cluster gets upgraded. The Pub/Sub uses pull subscription model. Now whenever a message is published to the pub/sub I need to set-up an alerting policy to pull the message and send an email containing the message content to a distribution channel via email. I need to achieve it without writing Cloud function only through alerting policy?
Can anyone please suggest how to achieve this? Thank you
Alert policy can't read the PubSub messages. The product listen to the logs and when the combination match a policy rule, an action (an alert) is generated.
If you need to send an email on the PubSub message content, you MUST read it (with Cloud Functions, Cloud Run, App Engine or whatever) and:
Either send directly the email with the message content
Or, if you want to use Cloud Alerting, publish a special log format (put a specific key word in the log that you write along to the message content), to let Cloud Alerting detect the log entries and send email alert with the log trace (including your message content)
I have to monitor CloudWatch logs for particular lambda. When the specific error message such as HTTP-50X is logged to CloudWatch, then we need to send an email notification to alert that the something is wrong.
We need help to create alert email in AWS to monitor manually the logs to see if there is any such error logged.
This is exactly what metric filter is for. Create a metric filter that monitors your log group for specific pattern. Then create a CloudWatch alarm based on a new metric and configure it with an SNS action. Then simply subscribe your email to that SNS topic.
All of this can be easily done via CloudFormation(Metric filter, Alarm, SNS). You can also use CDK.
You can process/monitor logs in real-time using lambda functions. Checkout Using AWS Lambda with Amazon CloudWatch Logs and Real-time Processing of Log Data with Subscriptions for details.
I got some alarm notifications from AWS Cloudwatch to my email but they are usually sent in a JSON format and the problem is that some of those emails are getting received by non-technical people in my company. I was wondering if that is possible to customize the emails sent by AWS SNS because I don't see any option in how to customize it.
If you are using Cloudwatch/Event/Rules then you can use "Input transfomer" to customize the contents of the email and then SNS will send that instead of JSON.
I am using this setup to get notified when user signs in AWS console.
In my case "Event Source" is "AWS Console Sign-in" but you can try with Cloudwatch and see if that gets you what you need.
See the screenshot, hope it helps.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatch-Events-Input-Transformer-Tutorial.html
https://forums.aws.amazon.com/thread.jspa?messageID=820808
you trigger a lambda function from your SNS topic, take the event and either send it via SNS or SES. SES supports sending HTML mails, SNS just plain text mails.
I normally do the following approach:
Alarm -> SNS -> Lambda -> SES (HTML)
received by non-technical people in my company
I'd use a basic HTML template with buttons and nice looking text :-)
I created a post and a GitHub repository for that:
https://medium.com/#sandro_volpicella/how-to-customise-cloudwatch-alarm-notifications-with-lambda-ses-html-and-cdk-f0094b07fed6
https://github.com/AlessandroVol23/cloudwatch-custom-email-cdk
I don't think you can change the notifications from SNS sent via e-mail.
If you really need to customize them, you should look into SNS notifications via SES (Simple Email Service)
I know that the AWS sends emails about any EC2 instances with Scheduled Events, but I would also like to have these notifications sent through HipChat and text message as well. Is there a default way of selecting a setting that can do this in AWS? Or would I have to create a more custom solution to achieve this?
With Scheduled Events in EC2 being manual tool for monitoring health of instances as of right now, there isn't many options. From the design point of view on AWS platform, this could be one possible solution:
Create AWS SES mailbox and enable it to receive emails,
Create a forwarding rule in your original e-mail inbox where you receive those e-mail notifications from AWS about Scheduled Events to forward e-mails of this type (based on filter you define) to the previously created AWS SES mailbox,
In your AWS SES mailbox create "Receipt Rule" with SNS Action, that will basically publish entire content of the e-mail to the SNS Topic using AWS SNS. (The SNS topic you choose must be in the same AWS region as the Amazon SES endpoint you use to receive email)
Once that e-mail content lands in AWS SNS Topic as an event, you can use AWS Lambda subscribed to your SNS topic as a handler of the event, parse what you need and forward customized notification to any of your third-party (outside of AWS) alerting tools (such as PagerDuty, Slack channel, HipChat, or whatever you use for SMS notifications).
This design does have one week point: relying onto your external mailbox forwarding system (if you are receiving Scheduled Events e-mails in inbox not based on AWS SES).
Ideally you could try to see in your AWS Account settings if it is possible to use AWS SES mailbox for Scheduled Events notifications, but I haven't tried that.
You can receive text messages as notification by setting alarm for that ec2 instance e.g if CPU usages goes above 80% then create an alarm and eventually you can set it to get emails or text messages. Thanks