Here's the scenario, after AWS Backup completes backup cluster aurora RDS, it will send 1 event message to SNS topic. Finally, I'll use lambda to get cluster information.
AWS Backup job completed -> Event message -> SNS topic -> trigger lambda
The thing is that I cannot find any information related to cluster ID ( tried with EventBridge ). I've tried to use Event Subscription ( RDS), but, it won't send any information while I perform AWS Backup.
Please let me know if you have any other ideas, thanks in advance
Related
Hope you all doing well, I am new with SQS and Cloudwatch and I need to create a Cloudwatch that would monitor SQS and would trigger Lambda with an event every time a message enters and every time a message is left.
On another note, the lambda function should scale up and down ASG service. So if anyone has a cookbook regarding those issues it would be very helpful
thank you so much!
It appears that your requirement is to scale Amazon EC2 instances when messages are waiting to be processed in an Amazon SQS queue.
The correct architecture for this would be to configure the Auto Scaling group to use a scaling policy based on the metric ApproximateNumberOfMessagesVisible. This is a metric that Amazon SQS queues send to Amazon CloudWatch Metrics. There is no need to use an AWS Lambda function.
For reference, see:
Scaling based on Amazon SQS - Amazon EC2 Auto Scaling
Rapid Auto Scaling with Amazon SQS | AWS News Blog
I am using various AWS services (Lambda, Glue, S3, Redshift, EC2) for ETL processing. I am trying to create a 'log ETL flow' to have monitoring and notifications sent out (email or otherwise) when a step fails in the process.
I have checked that each service I am using has metrics being logged in CloudWatch. I am now trying to figure out a smart way of processing this data in order to send out notifications when a step fails and/or have a central monitoring of the entire flow.
Are there any best practices or examples of this setup?
It seems to be the perfect case for the CloudWatch Alarms.
You can create a CloudWatch alarm that watches a single CloudWatch metric or the result of a math expression based on CloudWatch metrics. The alarm performs one or more actions based on the value of the metric or expression relative to a threshold over a number of time periods. The action can be an Amazon EC2 action, an Amazon EC2 Auto Scaling action, or a notification sent to an Amazon SNS topic.
You can create a chain CloudWatch Alarm -> SNS
You can either use SNS to notify users via SMS or Push Notifications.
Or you can do one step more SNS -> SES to deliver emails.
Hi came to know that I can enable AWS cloud watch alarm for AWS EC2 spot intance if there is intrputtion for termination notice, here is more details
Now CloudWatch users can setup a rule that automatically sends the EC2
Spot two-minute warning to an SNS topic to get a push notification.
I have no clue how to setup SNS topic to get intrputtion for termination notice?
Create an SNS topic and subscribe to this topic to get notifications.
Create a CloudWatch Event Rule to trigger an action whenever a spot instance is terminated.
Configure your event as shown in the screenshot below.
Select SNS Topic as the target and enter the ARN of the topic you created.
Is there a way to configure AWS, EMR, and SNS such that a message is published to an SNS topic when an EMR job fails? I am using a transient EMR cluster for a nightly ETL, and need to be notified if the process fails for any reason.
I needed the same functionality and what I did is I use the AWS data pipeline to create the cluster and send the SNS notification if the pipeline fails.
http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-emractivity.html
http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-snsalarm.html
I need to receive notifications whenver my instance in terminated. I know it can be done by cloudtrail and then using sns and sqs to get email for it, if you receive event of termination.
Is there a simpler way to do that ?
Any solution will is appreciated, but I prefer is doing using boto.
While it is not possible to receive a notification directly from Amazon EC2 when an instance is terminated, there are a couple of ways this could be accomplished:
Auto Scaling can send a notification when an instance managed by Auto Scaling is terminated. See: Configure Your Auto Scaling Group to Send Notifications
AWS Config can also be configured to send a Simple Notification Service (SNS) notification when resources change. This would send many notifications, so you would need to inspect and filter the notifications to find the one(s) indicating an instance termination. See the SNS reference in: Set Up AWS Config Using the Console and Example Amazon SNS Notification and Email from AWS Config.
Amazon Simple Notification Service (SNS) can also push a message to Amazon Queueing Service (SQS), which can be easily polled with the boto python SDK.
Receiving notifications via CloudTrail and CloudWatch Logs is somewhat messier, so I'd recommend the AWS Config method.
Now AWS introduced "rules" Under "Events" in AWS CloudWatch. In your case, you can select EC2 as Event Selector and SNS or SQS as Targets.
https://aws.amazon.com/blogs/aws/new-cloudwatch-events-track-and-respond-to-changes-to-your-aws-resources/
According to the AWS doc: Spot Instance Interruptions, it is possible to pool the instance-metadata in order to get an approximation of the termination time. You can build any custom monitoring solution around that.
> curl http://169.254.169.254/latest/meta-data/spot/instance-action
{"action": "stop", "time": "2017-09-18T08:22:00Z"}
If the instance is not scheduled for termination a http:400 will be returned.