Can we create a single AWS Lambda function to trigger the alarm for all unhealthy targets in all Target groups in an AWS Account ?
This link https://aws.amazon.com/blogs/networking-and-content-delivery/identifying-unhealthy-targets-of-elastic-load-balancer/ provides the information to create a single lambda function to monitor and trigger an alarm for single Target Group. I need to monitor and trigger alarms for multiple target groups using one lambda function and further I need the same lambda function to trigger the SNS to send the email. Can we achieve the same ?
Judging from blog post contents, it is achievable while using the proposed solution as a starting point. Although you would need to change a few things.
You will need to associate all of the alarms with the same SNS topic. Depending on the type of alarm you trigger, you will have different data available to you in incoming SNS message. To me the most logical would be to create UnHealthyHostCount alarm on target groups themselves
Lambda function code suggests that function was written with having only one target group for "AWS/ApplicationELB" and "AWS/NetworkELB" alarms in mind.
Remove this block:
else:
tg_arn = os.environ['TARGETGROUP_ARN'].strip()
tg_type = (os.environ['TARGETGROUP_TYPE'].strip()).lower()
Extract target group ARN (tg_arn) from alarm dimension TargetGroup from the incoming SNS
The rest should be more or less the same
Exact steps depend on your particular setup and goals, so treat this as a rough blueprint.
Related
I'm currently trying to monitor the EKS Node group status, sometimes my node groups show degraded and I want a CloudWatch alert whenever the status is in a Degraded state, I checked CloudWatch Metrics there are no standard metrics, and even I'm unable to find the event in Cloud trail,
Is there any possibility's to creating the alarm using AWS Cloud trail events, Event bridge, or CloudWatch
Kindly help to find the solution for this
For CloudWatch, please take a looks at this:
https://docs.aws.amazon.com/de_de/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html
I think you can combine Lambda & CloudWatch & EventBridge service here to implement your simple health-check status for a single or multiple node groups.
For your health check Lambda function:
We create a Lambda with Python3 (3.9 for example)
We describe the node group using Boto3
We put a custom metric to CloudWatch metrics so if the status is Active, we put 1 else 0.
When we have the function ready, we prepare the every 1 minutes (up to you) setup.
We create an EventBridge (EB) rule with every 1 min triggers
The EB rule destination is the Lambda function
Once we have enough data points from CloudWatch metrics, we can create a CloudWatch alarm to help us notifying to E-mail or others.
References:
https://stackify.com/custom-metrics-aws-lambda/
https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-run-lambda-schedule.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/eks.html#EKS.Client.describe_nodegroup
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudwatch.html
I have a Logger Lambda function that listens on a specific LogGroup and process specific log details.
I would like to attach the newly created LogGroups of other lambdas to that specific LogGroup, so it will process them as well - but since these other lambdas are created automatically, I need to do this automatically. Can I do it? How?
So there's no way to directly replicate the logs stored in a CloudWatch log group to another log group. You could do this by creating a subscription filter with a Lambda function to push the logs from each log group to the common one, but this would increase the costs for CloudWatch.
What I would suggest is either of the following:
Create a subscription filter for each of the log groups used by your Lambda functions to the common Lambda function so that it is triggered when logs are pushed to any of the log groups. This event can be set up after creating each function. Note, you would have to update the function policy of the common Lambda to allow it to be invoked from each log group (or just set up a wildcard).
Push all the logs for all of the functions to a single log group. This would take the least effort, but you would have to figure out how to effectively separate the logs per function (if that is required for your use case).
Is there any way to create a alarm in AWS for ip-addresses are about to running out?
Thanks!
There is no standard Amazon CloudWatch alarm for monitoring available IP addresses in a subnet. However, you could:
Create an AWS Lambda function that scans the subnet(s) and sends a custom metric to CloudWatch
Create an Amazon CloudWatch Events rule that triggers the Lambda function at a desired interval
Create an Alarm on the custom metric to notify you when the metric drops below a desired level
The benefit of this approach is that you will retain a history, within the custom metric, of the number of available IP addresses. However, it would involve a separate custom metric and alarm for each subnet.
Alternatively, you could configure the Lambda function to send a message to an Amazon SNS topic whenever there is a low quantity of IP addresses in a subnet. Then, subscribe to the SNS topic to receive notification. While there is no history available, it requires less configuration because it doesn't use CloudWatch.
I'm trying to find a way to make an Amazon EC2 instance stop automatically when a certain custom metric on CloudWatch passes a limit. So far if I've understood correctly based on these articles:
Discussion Forum: Custom Metric EC2 Action
CloudWatch Documentation: Create Alarms to Stop, Terminate, Reboot, or Recover an Instance
This will only work if the metric is defined as follows:
Tied to certain instance
With type of System/Linux
However in my case I have a custom metric that is actually not instance-related but "global" and if a certain limit is passed, I would need to stop all instances, no matter from which instance the limiting log is received.
Does anybody know if there is way to make this work? What I'd need is some way to make CloudWatch work like this:
If arbitrary custom metric value passes a certain limit -> stop defined instances not tied to the metric itself.
The main problem is that the EC2 option is greyed out as the metric is not tied to certain EC2 instance and I'm not sure if there's any way to do this without actually making the metric itself certain instance related.
Have the custom CloudWatch metric post alerts to an SNS topic.
Have the SNS topic trigger a Lambda function that shuts down your EC2 instances via a call to the AWS API.
I've got my web servers set up to autoscale at certain times of the day.
I can measure the load on the box using scripts executed by Consul - and this can trigger events at certain thresholds.
I want to push these two together and trigger autoscaling at certain load levels. (Assume CPU load at 75% is the threshold).
My question is: What is the mechanism to get load on a box to trigger an autoscaling group in AWS?
Assumptions:
I was not planning to use AWS Cloudwatch - but am interested if this is the solution.
I'm more interested in the autoscale triggering interface. Is it a queue or a rest endpoint?
As #mahdi said, you can easily use AWS Cloudwatch to do this.
However, if you want Consul (or anything outside the scope of an AWS "service") to do it, you can use lambda.
You would create a lambda function that scales your instance up or down (or both). Lambda can have many triggers, such as an HTTP endpoint through API Gateway. If you already have Consul set up to do it (sounds like you do since you said can trigger events at certain thresholds.) just make it issue the HTTP request to API Gateway to scale up or down.
You can create a CloudWatch alarm with a CPUUtilization metric and set it to change state when your instance has CPU utilization more than 75%. Then in the Auto Scaling Group, you use this alarm for scaling (in/out) policy. You can also control the number of instances in an Auto Scaling Group by manually (e.g. through your application running on one the instances) changing the Desired value. This documentation can be helpful.