I'm using python Boto3 code, when an instance is terminated from Auto Scaling group it notifies SNS which publishes the message to SQS. Lambda is also triggered when SNS is notified, which executes a boto script to grab the message from SQS.
I am using reference code from Sending and Receiving Messages in Amazon SQS.
Here is the code snippet:
if messages.get('Messages'):
m = messages.get('Messages')[0]
body = m['Body']
print('Received and deleted message: %s' % body)
The result is:
START RequestId: 1234-xxxxxxxx Version: $LATEST
{
"Type" : "Notification",
"MessageId" : "d1234xxxxxx",
"TopicArn" : "arn:aws:sns:us-east-1:xxxxxxxxxx:AutoScale-Topic",
"Subject" : "Auto Scaling: termination for group \"ASG\"",
"Message" : "{\"Progress\":50,\"AccountId\":\"xxxxxxxxx\",\"Description\":\"Terminating EC2 instance: i-123456\",\"RequestId\":\"db-xxxxx\",\"EndTime\":\"2017-07-13T22:17:19.678Z\",\"AutoScalingGroupARN\":\"arn:aws:autoscaling:us-east-1:360695249386:autoScalingGroup:fef71649-b184xxxxxx:autoScalingGroupName/ASG\",\"ActivityId\":\"db123xx\",\"EC2InstanceId\":\"i-123456\",\"StatusCode\"\"}",
"Timestamp" : "2017-07-",
"SignatureVersion" : "1",
"Signature" : "",
"SigningCertURL" : "https://sns.us-east-1.amazonaws.com/..",
"UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/
}
I only need EC2InstanceId of the terminated instance not the whole message. How can I extract the ID?
If your goal is to execute an AWS Lambda function (having the EC2 Instance ID as a parameter), there is no need to also publish the message to an Amazon SQS queue. In fact, this would be unreliable because you cannot guarantee that the message being retrieved from the SQS queue matches the invocation of your Lambda function.
Fortunately, when Auto Scaling sends an event to SNS and SNS then triggers a Lambda function, SNS passes the necessary information directly to the Lambda function.
Start your Lambda function with this code (or similar):
def lambda_handler(event, context):
# Dump the event to the log, for debugging purposes
print("Received event: " + json.dumps(event, indent=2))
# Extract the EC2 instance ID from the Auto Scaling event notification
message = event['Records'][0]['Sns']['Message']
autoscalingInfo = json.loads(message)
ec2InstanceId = autoscalingInfo['EC2InstanceId']
Your code then has the EC2 Instance ID, without having to use Amazon SQS.
The instance id is in the message. It's raw JSON, so you can parse it with the json package and get the information.
import json
if messages.get('Messages'):
m = messages.get('Messages')[0]
body = m['Body']
notification_message = json.loads(body["Message"])
print('instance id is: %s' % notification_message["EC2InstanceId"])
Related
I'm trying to create a SNS topic in AWS and subscribe a lambda function to it that will send notifications to Slack apps/users.
I did read this article -
https://aws.amazon.com/premiumsupport/knowledge-center/sns-lambda-webhooks-chime-slack-teams/
that describes how to do it using this lambda code:
#!/usr/bin/python3.6
import urllib3
import json
http = urllib3.PoolManager()
def lambda_handler(event, context):
url = "https://hooks.slack.com/services/xxxxxxx"
msg = {
"channel": "#CHANNEL_NAME",
"username": "WEBHOOK_USERNAME",
"text": event['Records'][0]['Sns']['Message'],
"icon_emoji": ""
}
encoded_msg = json.dumps(msg).encode('utf-8')
resp = http.request('POST',url, body=encoded_msg)
print({
"message": event['Records'][0]['Sns']['Message'],
"status_code": resp.status,
"response": resp.data
})
but the problem is, that in that implementation I have to create a lambda function for every user.
I want to subscribe multiple Slack apps/users to one SNS topic.
Is there a way of doing that without creating a lambda function for each one?
You really DON'T need Lambda. Just SNS and SLACK are enough.
I found a way to integrate AWS SNS with slack WITHOUT AWS Lambda or AWS chatbot. With this approach you can confirm the subscription easily.
Follow the video which show all the step clearly.
https://www.youtube.com/watch?v=CszzQcPAqNM
Steps to follow:
Create slack channel or use existing channel
Create a work flow with selecting Webhook
Create a variable name as "SubscribeURL". The name
is very important
Add the above variable in the message body of the
workflow Publish the workflow and get the url
Add the above Url as subscription of the SNS You will see the subscription URL in the
slack channel
Follow the URl and complete the subscription
Come back to the work flow and change the "SubscribeURL" variable to "Message"
The publish the
message in SNS. you will see the message in the slack channel.
Hi i would say you should go for a for loop and make a list of all the users. Either manually state them in the lambda or get them with api call from slack e.g. this one here: https://api.slack.com/methods/users.list
#!/usr/bin/python3.6
import urllib3
import json
http = urllib3.PoolManager()
def lambda_handler(event, context):
userlist = ["name1", "name2"]
for user in userlist:
url = "https://hooks.slack.com/services/xxxxxxx"
msg = {
"channel": "#" + user, # not sure if the hash has to be here
"username": "WEBHOOK_USERNAME",
"text": event['Records'][0]['Sns']['Message'],
"icon_emoji": ""
}
encoded_msg = json.dumps(msg).encode('utf-8')
resp = http.request('POST',url, body=encoded_msg)
print({
"message": event['Records'][0]['Sns']['Message'],
"status_code": resp.status,
"response": resp.data
})
Another solution you can do is set up email for the slack users, see link:
https://slack.com/help/articles/206819278-Send-emails-to-Slack
When you can just add the emails as subscribers to the sns topic. You can fileter the msg that the receiver gets with Subscription filter policy.
I have an AWS Lambda Function 'A' with a SQS DeadLetterQueue configured. When the Lambda fails to process an event, this is correctly sent to the DLQ. Is there a way to re-process events that ended into a DLQ?
I found two solution, but they both have drawbacks:
Create a new Lambda Function 'B' that reads from the SQS and then sends the events one by one to the previous Lambda 'A'. -> Here I have to write new code and deploy a new Function
Trigger again Lambda 'A' just when an event arrives in the SQS -> This looks dangerous as I can incur in looping executions
My ideal solution should be re-processing on demand the discarded events with Lambda 'A', without creating a new Lambda 'B' from scratch. Is there a way to accomplish this?
Finally, I didn't find any solution from AWS to reprocess the DLQ events of a Lambda Function. Then I created my own custom Lambda Function (I hope that this will be helpful to other developers with same issue):
import boto3
lamb = boto3.client('lambda')
sqs = boto3.resource('sqs')
queue = sqs.get_queue_by_name(QueueName='my_dlq_name')
def lambda_handler(event, context):
for _ in range(100):
messages_to_delete = []
for message in queue.receive_messages(MaxNumberOfMessages=10):
payload_bytes_array = bytes(message.body, encoding='utf8')
# print(payload_bytes_array)
lamb.invoke(
FunctionName='my_lambda_name',
InvocationType="Event", # Event = Invoke the function asynchronously.
Payload=payload_bytes_array
)
# Add message to delete
messages_to_delete.append({
'Id': message.message_id,
'ReceiptHandle': message.receipt_handle
})
# If you don't receive any notifications the messages_to_delete list will be empty
if len(messages_to_delete) == 0:
break
# Delete messages to remove them from SQS queue handle any errors
else:
deleted = queue.delete_messages(Entries=messages_to_delete)
print(deleted)
Part of the code is inspired by this post
I set up a CPU alarm for an EC2 instance that triggers an SNS Topic that has an endpoint that is a Lambda function. The Lambda function will then send ma an email and slack message telling me that an instance is in the alarm start and tell me exactly what instance it came from. I have the email and slack working and now I just need to get the instance ID from the event that my Lambda received from the alarm.
I get the following event in the Lambda function. I want to just pull out the instance ID from it, which in this example would be "i-07db9e2f61d100". It is located in "Dimensions".
How about also pulling out the "AlarmName" (which would be "cpu-mon" in this example)?
Here is all the data in the event I receive:
{'Records': [{'EventSource': 'aws:sns', 'EventVersion': '1.0', 'EventSubscriptionArn': 'arn:aws:sns:us-east-2:Alarm-test:db99f3fe-1c4b', 'Sns': {'Type': 'Notification', 'MessageId': '9921c85a-6f59-50c0', 'TopicArn': 'arn:aws:sns:us-east-2:4990:Alarm-test', 'Subject': 'ALARM: "cpu-mon" in US East (Ohio)', 'Message': '{"AlarmName":"cpu-mon","AlarmDescription":"Alarm when CPU exceeds 70 percent","AWSAccountId":"000000000","NewStateValue":"ALARM","NewStateReason":"Threshold Crossed: 2 out of the last 2 datapoints [99.8333333333333 (26/08/19 19:19:00), 99.1803278688525 (26/08/19 19:18:00)] were greater than the threshold (70.0) (minimum 2 datapoints for OK -> ALARM transition).","StateChangeTime":"2019-08-26T19:20:52.350+0000","Region":"US East (Ohio)","OldStateValue":"OK","Trigger":{"MetricName":"CPUUtilization","Namespace":"AWS/EC2","StatisticType":"Statistic","Statistic":"AVERAGE","Unit":"Percent","Dimensions":[{"value":"i-07db9e2f61d100","name":"InstanceId"}],"Period":60,"EvaluationPeriods":2,"ComparisonOperator":"GreaterThanThreshold","Threshold":70.0,"TreatMissingData":"","EvaluateLowSampleCountPercentile":""}}', 'Timestamp': '2019-08-26T19:20:52.403Z', 'SignatureVersion': '1', 'Signature': 'UeWhS==', 'SigningCertUrl': 'https://sns.us-east-2.amazonaws.com/SimpleNotificationService-63f9.pem', 'UnsubscribeUrl': 'https://sns.us-east-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-2:49:Alarm-test:dfe-1c4b-4db9', 'MessageAttributes': {}}}]}
Here is my Lambda function (python) -
# Sends Slack and text message
import json
import subprocess
import boto3
session = boto3.Session(
region_name="us-east-1"
)
sns_client = session.client('sns')
def lambda_handler(event, context):
print("THIS IS THE EVENT - " + str(event))
data = json.dumps({'text': str(event)})
# Send text alerts
alertNumbers = ["1-xxx-xxx-xxxx"]
# Send text message
for i in range(len(alertNumbers)):
sns_client.publish(
PhoneNumber=alertNumbers[i],
Message=msg,
MessageAttributes={
'AWS.SNS.SMS.SenderID': {
'DataType': 'String',
'StringValue': 'SENDERID'
},
'AWS.SNS.SMS.SMSType': {
'DataType': 'String',
'StringValue': 'Promotional'
}
}
)
# Send Slack message
subprocess.call([
'curl',
'-X', 'POST',
'-H', 'Content-type: application/json',
'--data', data,
'https://hooks.slack.com/services/000000'
Thanks for any help!
You simply need to access the data of the event and put it where you want it.
Inside your lambda_handler add this as the first line:
message = json.loads(event['Records'][0]['Sns']['Message'])
Now the SNS message is available as message. To get the AlarmName is as simple as message['AlarmName'] and the instance id is at message['Trigger']['Dimensions'][0]['value']
I'd like to send SNS notifications into Slack. I receive notifications on my e-mail. It looks like:
Instance: i-0f9606e41cd6f1e8e has changed state
State: running
Type: c5.4xlarge
Public IP Address: 52.32.193.26
Private IP Address: 10.10.75.168
Region: us-west-2a
Name: VOSaaS-Cluster-SaaS-Longevity-055ba27d-f7c4-b70a-0954-a08ae21ccb2d-vos-node-i-0f9606e41cd6f1e8e
But also I want to receive the same output into my Slack channel. I've already set up the incoming webhooks and I can receive simple messages but have a problem with sending output.
MY_SNS_TOPIC_ARN = 'arn:aws:sns:us-west-2:421572644019:CloudWatchAlarmsForSpotInstances'
sns_client = boto3.client('sns')
ec2_spot_info = sns_client.publish(
TopicArn = MY_SNS_TOPIC_ARN,
Subject = 'EC2 Spot Instances Termination Notifications',
Message = 'Instance: ' + instance_id + ' has changed state\n' +
'State: ' + instance['State']['Name'] + '\n' +
'Type: ' + instance['InstanceType'] + '\n' +
'Public IP Address: ' + instance['PublicIpAddress'] + '\n' +
'Private IP Address: ' + instance['PrivateIpAddress'] + '\n' +
'Region: ' + instance['Placement']['AvailabilityZone'] + '\n' +
'Name: ' + name
)
slack_url='https://hooks.slack.com/services/+token'
slack_msg = {
"attachments": [
{
"title": "EC2 Spot Instance Info",
"pretext": "EC2 Spot Instances Termination Notifications",
"color": "#ed1717",
"text": ec2_spot_info
}
]
}
output = json.dumps(slack_msg)
r = requests.post(slack_url, data = output)
The sns_client.publish() call returns a response of:
{
'MessageId': 'string'
}
Yet your slack command is sending this as a message:
"text": ec2_spot_info
This means that, instead of sending a message to slack, you are sending a dictionary containing the MessageId.
Instead, you should:
Construct message as a variable
Call sns_client.publish() with Message = message
Call slack using "text": message
There is an issue when When you Subscribe Slack wehbook with SNS.
The slack is unable to convert/read the payload comes from SNS. You have to do a bit of hack to read the SubscribeURL/Message.
Try with pure SNS topic with a slack channel first.
You can use SLACK workflow with SNS.
Follow the video which show all the steps clearly.
https://www.youtube.com/watch?v=CszzQcPAqN
Steps to follow:
Create slack channel or use existing channel
Create a work flow with selecting Webhook
Create a variable name as "SubscribeURL". The name is very important
Add the above variable in the message body of the workflow
Publish the workflow and get the url
Add the above Url as subscription of the SNS
You will see the subscription URL in the slack channel
Follow the URl and complete the subscription
Come back to the work flow and change the variable to "Message"
The publish the message in SNS. you will see the message in the slack channel.
I need to catch the event RunflowJob in my Cloudwatch EventRule in order to tag AWS EMR starting clusters.
I'm looking for this event, because i need the username and account informations
Any idea?
Thanks
Calls to the ListClusters, DescribeCluster, and RunJobFlow actions generate entries in CloudTrail log files.
Every log entry contains information about who generated the request. For example, if a request is made to create and run a new job flow (RunJobFlow), CloudTrail logs the user identity of the person or service that made the request
https://docs.aws.amazon.com/emr/latest/ManagementGuide/logging_emr_api_calls.html#understanding_emr_log_file_entries
Here is a sample snippet to get the username using Python Boto3.
import boto3
cloudtrail = boto3.client("cloudtrail")
response = cloudtrail.lookup_events (
LookupAttributes=[
{
'AttributeKey': 'EventName',
'AttributeValue': 'RunJobFlow'
}
],
)
for event in response.get ("Events"):
print(event.get ("Username"))
username and cluster details can be retrieved from the RunJobFlow event itself. Easier solution would be to use Cloudwatch event rule along with Lambda function as a target to fetch these info and subsequently further action can be taken as required. Example below:
Event Pattern to be used with Cloudwatch event rule
{
"source": ["aws.elasticmapreduce"],
"detail": {
"eventName": ["RunJobFlow"]
}
}
Lambda code snippet
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
user = event['detail']['userIdentity']['userName']
cluster_id = event['detail']['responseElements']['jobFlowId']
region = event['region']