I want to receive notifications from AWS Eventbridge when there's a scheduled event for my Amazon Elastic Compute Cloud (Amazon EC2) instance.
I created an Eventbridge rule and set the target to an already working SNS topic. The SNS topic is subscribed to a working Lambda function that is used for other "Cloudwatch to slack" alarms already. The eventbridge setting is as follows:
{
"source": ["aws.health"],
"detail-type": ["AWS Health Event"],
"detail": {
"service": ["EC2"],
"eventTypeCategory": ["scheduledChange"]
}
}
I already got an EC2 scheduled maintenance(reboot) notification as e-mail from AWS, but this eventbridge I created did not trigger for that and did not send any notification to the slack channel.
I am unsure now if I am missing something in the setting. I am setting it for the first time and no way to simply test it with fake input. It is supposed to work even if there is a single schedule event that appears in the top bell icon(as shown in the screenshot above), correct?
In order to find out the root cause of this issue, I suggest to take a look a the CloudWatch usage metrics for SNS. SNS reports the following metrics which might be useful for you: NumberOfMessagesPublished, NumberOfNotificationsDelivered NumberOfNotificationsFailed. If you find these metrics reported an they have a value different than 0, this means that SNS receives events from Event Bridge and the problem is somewhere else.
If you are using a Lambda to send messages to Slack, you should take a look at the logs in CloudWatch to see if the Lambda did execute successfully. You might want to check out the setup for Lambda recommended by AWS: (link)
For further debugging you may want to check out test-event-pattern CLI command.
It is supposed to work even if there is a single schedule event that appears in the top bell icon(as shown in the screenshot above), correct?
Yeah, it supposed to work even if there already is an event.
I'm having a similar issue with eventbridge rule being built with cloudformation. I had to manually go into the eventbridge rule via the AWS console and go to the trigger and select the SNS topic again. It now works. It took me a while to figure out. Can you confirm that the fix did that for you as I'm not sure how to fix this...
Related
I can see that in AWS alerting works with SNS where protocols are available as Email, https and SMS...Basically push mechanisms..
Is it possible to make these alerts pull-based? any ideas around how this could be tried?
You would need to create a solution for this, natively when an alert triggers it can be configured to go to either an SNS topic which has 1 or more subscriptions to notify.
Instead you would need to use the describe-alarms function to retrieve a list of alarms, this can be narrowed down to alarming only by using the --state-value parameter such as the example below
aws cloudwatch describe-alarms --state-value ALARM
Does this fit your needs https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudwatch/describe-alarm-history.html ?
Please check aws cloudwatch there are several options for alarms.
I have to monitor a website if it works - response looks like following:
{"response":
{"time": 1457564305},
"stat": "OK"
}
And need sending mail alert to me if stat is not OK.
Seems this can be done with Lambda using node.js and CloudWatch.
I tried to create Lambda function and execute through CloudWatch but only see the function was triggered. Seems I need to push some metrics?
I'm newbie to node.js and Lambda (handler, metrics, etc.).
Would anyone share some hints on where I should start with?
Thanks a lot.
Not sure what you mean by "only see the function was triggered".
Yes, you can cause a Lambda function to be invoked on a schedule. That Lambda function could issue a request to the website and trigger a notification to an SNS topic if it believes the site to be down. You would subscribe (by email address or SMS) to the SNS topic for notifications. Here's an example of how to do this.
I would like to create a cloudwatch alarm for the sns metric NumberOfMessagesPublished on a new sns topic. If I attempt to do this before I trigger the topic (and there is no data for it) then the option to create an alarm does not exist in the web console.
How can I do this with the web console? Is using a cli tool the only option?
I noticed that too -- you cannot select a metric until the data has been sent to CloudWatch.
I suspect that CloudWatch actually doesn't have any visibility into metrics until they are sent from the originating service into CloudWatch. Thus, it can't populate the console because it doesn't know what data will be sent. That would be a nice, loosely-coupled design.
My experiments show that it is possible create an alarm via the AWS Command-Line Interface (CLI) on metrics that do not exist -- even within the EC2 namespace. That's the way to go!
I suggest you to use Cloudformation. It is the best way to create resources stack in AWS and overall you can easily experiment with it and delete everything when you are done. Here there is a sample application that uses Cloudformation and Cloudwatch https://github.com/awslabs/cloudwatch-dashboards-cloudformation-sample
The manual solution I went with was to create the sns topic and then use the Publish to topic option in the sns menu to send out a test message to my subscribers. This creates a set of notifications without the need to set off the cloudwatch rule. After this the option appears in the cloudwatch alarm creation menu.
The other CLI options would be better for automation.
Yes, it is possible. The AWS Official Documentation answers your question. Just to quote the relevant lines:
You can create an alarm for a custom metric before you've created that custom metric. For the alarm to be valid, you must include all of the dimensions for the custom metric in addition to the metric namespace and metric name in the alarm definition.
I have a AWS Lambda function running some process in my infrastructure. The Lambda is triggered every 8 hours using a CloudWatch rule. I am trying to raise a notification if any error happens into the Lambda process. I tried to use SES but that service is not available in that Region.
I will like to know any suggestions for this problem:
How to setup notifications when an error occurs in my Lambda functions ?
I am looking for suggestions. This questions never asked for doing my task. I will appreciate any official documentation but either way, any help is welcome.
Some suggestions:
Dead Letter Queues:
If your error causes failed invocations, you can use a Lambda Dead Letter Queue to send the event to an SNS topic or an SQS queue. If you send it to an SNS topic, you can directly subscribe to the topic via SNS or Email to get notified any time a message is published to that topic.
Multi-region SES:
If you're really set on using SES directly, SES clients can be instantiated with an explicit region provided -- as long as your lambda's execution role has the appropriate permissions, you can send email to SES from a different region. Here's documentation for instantiating the JS SES Client.
CloudWatch Logs:
If your error does not cause the invocation to fail, another option is using a CloudWatch Logs metric filter to aggregate failures and potentially alarm on them. If you're using NodeJS, you can simply log out via console.log(), console.error(), etc. and it will be written out to CWLogs. More details here.
You can subscribe an SNS topic to CloudWatch Alarms, and notify yourself in the same way as the DLQ.
As you gain experience with the error and learn how to process common errors, you could also subscribe another lambda to the SNS topic from the DLQ/CWLogs example to process it as it happens.
I need to receive notifications whenver my instance in terminated. I know it can be done by cloudtrail and then using sns and sqs to get email for it, if you receive event of termination.
Is there a simpler way to do that ?
Any solution will is appreciated, but I prefer is doing using boto.
While it is not possible to receive a notification directly from Amazon EC2 when an instance is terminated, there are a couple of ways this could be accomplished:
Auto Scaling can send a notification when an instance managed by Auto Scaling is terminated. See: Configure Your Auto Scaling Group to Send Notifications
AWS Config can also be configured to send a Simple Notification Service (SNS) notification when resources change. This would send many notifications, so you would need to inspect and filter the notifications to find the one(s) indicating an instance termination. See the SNS reference in: Set Up AWS Config Using the Console and Example Amazon SNS Notification and Email from AWS Config.
Amazon Simple Notification Service (SNS) can also push a message to Amazon Queueing Service (SQS), which can be easily polled with the boto python SDK.
Receiving notifications via CloudTrail and CloudWatch Logs is somewhat messier, so I'd recommend the AWS Config method.
Now AWS introduced "rules" Under "Events" in AWS CloudWatch. In your case, you can select EC2 as Event Selector and SNS or SQS as Targets.
https://aws.amazon.com/blogs/aws/new-cloudwatch-events-track-and-respond-to-changes-to-your-aws-resources/
According to the AWS doc: Spot Instance Interruptions, it is possible to pool the instance-metadata in order to get an approximation of the termination time. You can build any custom monitoring solution around that.
> curl http://169.254.169.254/latest/meta-data/spot/instance-action
{"action": "stop", "time": "2017-09-18T08:22:00Z"}
If the instance is not scheduled for termination a http:400 will be returned.