SQS: Publish a message when an EC2 instance starts? - amazon-web-services

Is there a way to automatically publish a message on SQS when an EC2 instance starts?
For example, could you use Cloudwatch to set an alarm that fires whenever an instance starts up? All the alarms on Cloudwatch appear to be related to a specific EC2 instance, rather than the EC2 service in general.

To better understand this question and offer a more accurate answer, further information is needed.
Are we talking about:
New instance created and started from any AMI ?
New instance created and started from a specific AMI?
Starting an existing AMI that is just in the stopped state?
Or creating a new instance inside a scale group?
These all affect the way you would create your cloudwatch alarm.
For example if it were an existing ec2 you would use status checks as per:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html
Though if it were a brand new Ec2 instance created you would need to use more advanced cloudtrail log alarms as per:
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cw_create_alarms.html
However after that point it would follow the same basic logic and that is:
Create an Alarm that triggers a SNS as per:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/ConsoleAlarms.html
have that SNS Notifier publish to a SQS topic as per:
https://docs.aws.amazon.com/sns/latest/dg/SendMessageToSQS.html
As always though there are many ways to skin a cat.
If it were a critical event and I want the same typical response from every start event I personally would consider bootstrap scripts pushed out from puppet or chef, this way a fast change for all events is centralised in a single script.

Step1 : Create a Cloud watch Rule to notifiy the creation . As per the lifecycle of EC2 Instance when lauch button is pressed . Instance goes from Pending state to Running state and If instance is moved from stop state to running state. So create the Rule for Pending state
Create a Cloud watch Rule as specified in the image screenshot
Step2 : Create a Step function . Because cloud Trail logs all the event in the account with a delay of atleast 20 min . This step function is usefull if you want the name of user who has created the instance .
{
"StartAt": "Wait",
"States": {
"Wait": {
"Type": "Wait",
"Seconds": 1800,
"Next": "Ec2-Alert"
},
"Ec2-Alert":{
"Type": "Task",
"Resource":"arn:aws:lambda:ap-south-1:321039853697:function:EC2-Creation-Alert",
"End": true
}
}
}
Step3 : Create a SNS topic for notification
Step4 : Write a lambda function to fetch the log from cloud trail and get the user name who has created the instance .
import json
import os
import subprocess
import boto3
def lambda_handler(event, context):
client = boto3.client('cloudtrail')
client1 = boto3.client('sns')
Instance=event["detail"]["instance-id"]
response = client.lookup_events(
LookupAttributes=[
{
'AttributeKey': 'ResourceName',
'AttributeValue': Instance
},
],
MaxResults=1)
test=response['Events']
st="".join(str(x) for x in test)
print(st)
user=st.split("Username")[1]
finalname=user.split(",")
Creator=finalname[0]
#print(st[st.find("Username")])
Email= "Hi All ,\n\n\n The User%s has created new EC2-Instance in QA account and the Instance id is %s \n\n\n Thank you \n\n\n Regard's lamda"%(Creator,Instance)
response = client1.publish(
TopicArn='arn:aws:sns:ap-south-1:321039853697:Ec2-Creation-Alert',
Message=Email
)
# TODO implement
return {
'statusCode': 200,
}
Note: This code trigger an notification if the instance is changed from stop state to running state or a new instance is launched .

Related

How to get the event content in ECS when it is invoked by cloudwatch/eventbridge event?

We can set up event rules to trigger an ECS task, but I don't see if the triggering event is passed to the runing ECS task and in the task how to fetch the content of this event. If a Lambda is triggered, we can get it from the event variable, for example, in Python:
def lambda_handler(event, context):
...
But in ECS I don't see how I can do things similar. Going to the cloudtrail log bucket doesn't sound to be a good way because it has around 5 mins delay for the new log/event to show up, which requires ECS to be waiting and additional logic to talk to S3 and find & read the log. And when the triggering events are frequent, this sounds hard to handle.
One way to handle this is to set two targets In the Cloud watch rule.
One target will launch the ECS task
One target will push same event to SQS
So the SQS will contain info like
{
"version": "0",
"id": "89d1a02d-5ec7-412e-82f5-13505f849b41",
"detail-type": "Scheduled Event",
"source": "aws.events",
"account": "123456789012",
"time": "2016-12-30T18:44:49Z",
"region": "us-east-1",
"resources": [
"arn:aws:events:us-east-1:123456789012:rule/SampleRule"
],
"detail": {}
}
So when the ECS TASK up, it will be able to read event from the SQS.
For example in Docker entrypoint
#!/bin/sh
echo "Starting container"
echo "Process SQS event"
node process_schdule_event.sj
#or if you need process at run time
schdule_event=$(aws sqs receive-message --queue-url https://sqs.us-west-2.amazonaws.com/123456789/demo --attribute-names All --message-attribute-names All --max-number-of-messages 1)
echo "Schdule Event: ${schdule_event}"
# one process done, start the main process of the container
exec "$#"
After further investigation, I finally worked out another solution that is to use S3 to invoke Lambda and then in that Lambda I use ECS SDK (boto3, I use Python) to run my ECS task. By this way I can easily pass the event content to ECS and it is nearly real-time.
But I still give credit to #Adiii because his solution also works.

Send SNS Notification when EC2 instance is rebooted

I need help with one AWS Step. I know we can send an SNS Notification when the Instance is Stopped, terminated, starting and pending stages. But how do we send a notification when the EC2 instance is rebooted?
Thanks!
If a reboot is issued within the instance, then this will not be detected by AWS. It is just the operating system doing its own things.
If a reboot is issued through the AWS management console or an API call, the instance does not actually change state. From Instance Lifecycle - Amazon Elastic Compute Cloud:
Rebooting an instance is equivalent to rebooting an operating system. The instance remains on the same host computer and maintains its public DNS name, private IP address, and any data on its instance store volumes. It typically takes a few minutes for the reboot to complete, but the time it takes to reboot depends on the instance configuration.
Therefore, the only way to react to the reboot command being issued via the AWS console or API is to create an AWS CloudWatch Events rule that receives all Amazon EC2 events and then checks whether it is specifically for a RebootInstances command.
The rule would look like:
{
"source": [
"aws.ec2"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"ec2.amazonaws.com"
],
"eventName": [
"RebootInstances"
]
}
}
It can then trigger an Amazon SNS notification, which will include the instanceId.
However, the notification is not very pretty — it consists of a blob of JSON. If you want to send a nicer-looking message, see: amazon web services - Email notification through SNS and Lambda - Stack Overflow
Instead of monitoring Cloudtrail you could create a cron entry on instance that will execute #reboot (example here) and send sns notification using aws cli sns publish.
You can use the crontab with #reboot against a script to run.
$ crontab -e
#reboot $(python /home/ec2-user/sms.py)
for the python script sms.py at /home/ec2-user
sms.py (change the AWS region if required):
import boto3
import json
client = boto3.client('sns', region_name='us-west-2')
msg = 'Instance reboot!'
response = client.publish(
PhoneNumber='+1XXXXXXXXXX',
Message=msg)
print(response)
Make sure boto3 is installed on the instance - $ pip install boto3 --user

How to test lambda using test event

I have lambda which is triggered by cloudwatch event when VPN tunnels are down or up. I searched online but can't find a way to trigger this cloudwatch event.
I see an option for test event but what can I enter in here for it to trigger an event that tunnel is up or down?
You can look into CloudWatchEventsandEventPatterns
Events in Amazon CloudWatch Events are represented as JSON objects.
For more information about JSON objects, see RFC 7159. The following
is an example event:
{
"version": "0",
"id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
"detail-type": "EC2 Instance State-change Notification",
"source": "aws.ec2",
"account": "111122223333",
"time": "2017-12-22T18:43:48Z",
"region": "us-west-1",
"resources": [
"arn:aws:ec2:us-west-1:123456789012:instance/ i-1234567890abcdef0"
],
"detail": {
"instance-id": " i-1234567890abcdef0",
"state": "terminated"
}
}
Also log based on event, you can pick your required event from AWS CW EventTypes
I believe in your scenario, you don't need to pass any input data as you must have built the logic to test the VPN tunnels connectivity within the Lamda. You can remove that JSON from the test event and then run the test.
If you need to pass in some information as part of the input event then follow the approach mentioned by #Adiii.
EDIT
The question is more clear through the comment which says
But question is how will I trigger the lambda? Lets say I want to
trigger it when tunnel is down? How will let lambda know tunnel is in
down state? – NoviceMe
This can be achieved by setting up a rule in Cloudwatch to schedule the lambda trigger at a periodic interval. More details here:
Tutorial: Schedule AWS Lambda Functions Using CloudWatch Events
Lambda does not have an invocation trigger right now that can monitor a VPN tunnel, so the only workaround is to poll the status through lamda.

AWS trigger when an instance is available

I have created a workflow like this:
Use requests for an instance creation through a API Gateway endpoint
The gateway invokes a lamda function that executes the following code
Generate a RDP with the public dns to give it to the user so that they can connect.
ec2 = boto3.resource('ec2', region_name='us-east-1')
instances = ec2.create_instances(...)
instance = instances[0]
time.sleep(3)
instance.load()
return instance.public_dns_name
The problem with this approach is that the user has to wait almost 2 minutes before they were able to login successfully. I'm totally okay to let the lamda run for that time by adding the following code:
instance.wait_until_running()
But unfortunately the API gateway has a 29 seconds timeout for lambda integration. So even if I'm willing to spend it wouldn't work. What's the easiest way to overcome this?
My approach to accomplish your scenario could be Cloudwatch Event Rule.
The lambda function after Instance creation must store a kind of relation between the instance and user, something like this:
Proposed table:
The table structure is up to you, but these are the most important columns.
------------------------------
| Instance_id | User_Id |
------------------------------
Creates a CloudWatch Event Rule to execute a Lambda function.
Firstly, pick Event Type: EC2 Instance State-change Notification then select Specific state(s): Running:
Secondly, pick the target: Lambda function:
Lambda Function to send email to the user.
That Lambda function will receive the InstanceId. With that information, you can find the related User_Id and send the necessary information to the user. You can use the SDK to get information of your EC2 instance, for example, its public_dns_name.
This is an example of the payload that will be sent by Cloudwatch Event Rule notification:
{
"version": "0",
"id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
"detail-type": "EC2 Instance State-change Notification",
"source": "aws.ec2",
"account": "111122223333",
"time": "2015-12-22T18:43:48Z",
"region": "us-east-1",
"resources": [
"arn:aws:ec2:us-east-1:123456789012:instance/i-12345678"
],
"detail": {
"instance-id": "i-12345678",
"state": "running"
}
}
That way, you can send the public_dns_name when your instance is totally in service.
Hope it helps!

Launch a shell script from Lambda in AWS

If I have a bash script sitting in an EC2 instance, is there a way that lambda could trigger it?
The trigger for lambda would be coming from RDS. So a table in mysql gets updated and a specific column in that table gets updated to "Ready", Lambda would have to pull the ID of that row with a "Ready" status and send that ID to the bash script.
Let's assume some things. First, you know how to set up a "trigger" using sns (see here) and how to hang a lambda script off of said trigger. Secondly, you know a little about python (Lambda's syntax offerings are Node, Java, and Python) because this example will be in Python. Additionally, I will not cover how to query a database with mysql. You did not mention whether your RDS instance was MySQL, Postgress, or otherwise. Lastly, you need to understand how to allow permission across AWS resources with IAM roles and policies.
The following script will at least outline the method of firing a script to your instance (you'll have to figure out how to query for relevant information or pass that information into the SNS topic), and then run the shell command on an instance you specify.
import boto3
def lambda_handler(event, context):
#query RDS to get ID or get from SNS topic
id = *queryresult*
command = 'sh /path/to/scriptoninstance' + id
ssm = boto3.client('ssm')
ssmresponse = ssm.send_command(InstanceIds=['i-instanceid'], DocumentName='AWS-RunShellScript', Parameters= { 'commands': [command] } )
I would probably have two flags for the RDS row. One that says 'ready' and one that says 'identified'. So SNS topic triggers lambda script, lambda script looks for rows with 'ready' = true and 'identified' = false, change 'identified' to true (to make sure other lambda scripts that could be running at the same time aren't going to pick it up), then fire script. If script doesn't run successfully, change 'identified' back to false to make sure your data stays valid.
Using Amazon EC2 Simple Systems Manager, you can configure an SSM document to run a script on an instance, and pass that script a parameter. The Lambda instance would need to run the SSM send-command, targeting the instance by its instance id.
Sample SSM document:
run_my_example.json:
{
"schemaVersion": "1.2",
"description": "Run shell script to launch.",
"parameters": {
"taskId":{
"type":"String",
"default":"",
"description":"(Required) the Id of the task to run",
"maxChars":16
}
},
"runtimeConfig": {
"aws:runShellScript": {
"properties": [
{
"id": "0.aws:runShellScript",
"runCommand": ["run_my_example.sh"]
}
]
}
}
}
The above SSM document accepts taskId as a parameter.
Save this document as a JSON file, and call create-document using the AWS CLI:
aws ssm create-document --content file:///tmp/run_my_example.json --name "run_my_example"
You can review the description of the SSM document by calling describe-document:
aws ssm describe-document --name "run_my_example"
You can specify the taskId parameter and run the command by using the document name with the send-command
aws ssm send-command --instance-ids i-12345678 --document-name "run_my_example" --parameters --taskid=123456
NOTES
Instances must be running the latest version of the SSM agent.
You will need to have some logic in the Lambda script to identify the instance ids of the server EG look up the instance id of a specifically tagged instance.
I think you could use the new EC2 Run Command feature to accomplish this.
There are few things to consider one of them is:
Security. As of today lambda can't run in VPC. Which means your EC2 has to have a wide open inbound security group.
I would suggest take a look at the messaging queue (say SQS ). This would solve a lot of headache.
That's how it might work:
Lambda. Get message; send to SQS
EC2. Cron job that gets trigger N number of minutes. pull message from sqs; process message.