Launch a shell script from Lambda in AWS - amazon-web-services

If I have a bash script sitting in an EC2 instance, is there a way that lambda could trigger it?
The trigger for lambda would be coming from RDS. So a table in mysql gets updated and a specific column in that table gets updated to "Ready", Lambda would have to pull the ID of that row with a "Ready" status and send that ID to the bash script.

Let's assume some things. First, you know how to set up a "trigger" using sns (see here) and how to hang a lambda script off of said trigger. Secondly, you know a little about python (Lambda's syntax offerings are Node, Java, and Python) because this example will be in Python. Additionally, I will not cover how to query a database with mysql. You did not mention whether your RDS instance was MySQL, Postgress, or otherwise. Lastly, you need to understand how to allow permission across AWS resources with IAM roles and policies.
The following script will at least outline the method of firing a script to your instance (you'll have to figure out how to query for relevant information or pass that information into the SNS topic), and then run the shell command on an instance you specify.
import boto3
def lambda_handler(event, context):
#query RDS to get ID or get from SNS topic
id = *queryresult*
command = 'sh /path/to/scriptoninstance' + id
ssm = boto3.client('ssm')
ssmresponse = ssm.send_command(InstanceIds=['i-instanceid'], DocumentName='AWS-RunShellScript', Parameters= { 'commands': [command] } )
I would probably have two flags for the RDS row. One that says 'ready' and one that says 'identified'. So SNS topic triggers lambda script, lambda script looks for rows with 'ready' = true and 'identified' = false, change 'identified' to true (to make sure other lambda scripts that could be running at the same time aren't going to pick it up), then fire script. If script doesn't run successfully, change 'identified' back to false to make sure your data stays valid.

Using Amazon EC2 Simple Systems Manager, you can configure an SSM document to run a script on an instance, and pass that script a parameter. The Lambda instance would need to run the SSM send-command, targeting the instance by its instance id.
Sample SSM document:
run_my_example.json:
{
"schemaVersion": "1.2",
"description": "Run shell script to launch.",
"parameters": {
"taskId":{
"type":"String",
"default":"",
"description":"(Required) the Id of the task to run",
"maxChars":16
}
},
"runtimeConfig": {
"aws:runShellScript": {
"properties": [
{
"id": "0.aws:runShellScript",
"runCommand": ["run_my_example.sh"]
}
]
}
}
}
The above SSM document accepts taskId as a parameter.
Save this document as a JSON file, and call create-document using the AWS CLI:
aws ssm create-document --content file:///tmp/run_my_example.json --name "run_my_example"
You can review the description of the SSM document by calling describe-document:
aws ssm describe-document --name "run_my_example"
You can specify the taskId parameter and run the command by using the document name with the send-command
aws ssm send-command --instance-ids i-12345678 --document-name "run_my_example" --parameters --taskid=123456
NOTES
Instances must be running the latest version of the SSM agent.
You will need to have some logic in the Lambda script to identify the instance ids of the server EG look up the instance id of a specifically tagged instance.

I think you could use the new EC2 Run Command feature to accomplish this.

There are few things to consider one of them is:
Security. As of today lambda can't run in VPC. Which means your EC2 has to have a wide open inbound security group.
I would suggest take a look at the messaging queue (say SQS ). This would solve a lot of headache.
That's how it might work:
Lambda. Get message; send to SQS
EC2. Cron job that gets trigger N number of minutes. pull message from sqs; process message.

Related

How to add Cloudwatch output to runShellScript document on an EC2?

I have a python script on an EC2 which needs to run daily without anyone manually kicking it off. My current setup uses a scheduled Lambda function to send an SSM Document as a command to the EC2. The SSM Document includes a short "runShellScript" command to run the python script. (see below for the SSM document and abbreviated lambda function). This process works fine.
The issue is that I need the logs to stream to CloudWatch. I'm aware that CloudWatch can retrieve log files which are sitting on the EC2; However I want Cloudwatch to capture the logs directly from stdout (standard-out), rather than taking the log files.
When I manually run the SSM Document via the "Run Command" section of the AWS UI, it sends it to Cloudwatch beautifully since I directly configure the CloudWatch as part of the Run Command kickoff. However I don't see anywhere to configure Cloudwatch as part of the Document.
How can I adjust my SSM Document (or any piece of this process) to stream the logs to CloudWatch?
I'm open to changing schemaVersions in the document if that would help. I already looked through the SSM Parameter documentation for this but could not find an answer there.
Here is the relevant section of the Lambda function:
def lambda_handler(event, context):
# Execute the script
ssm = boto3.client('ssm', region_name=region)
ssm_response = ssm.send_command(InstanceIds=instances, DocumentName='CustomRunScript', Comment='Starting init script from lambda prod')
print('SSM response is: ', ssm_response)
Here is my SSM Document:
{
"schemaVersion": "1.2",
"description": "Custom Run Script",
"parameters": {},
"runtimeConfig": {
"aws:runShellScript": {
"properties": [
{
"id": "0.aws:runShellScript",
"runCommand": [
"/usr/bin/python3 /home/app/init.py"
]
}
]
}
}
}
I think you're looking for CloudWatchOutputConfig.
def lambda_handler(event, context):
# Execute the script
ssm = boto3.client('ssm', region_name=region)
ssm_response = ssm.send_command(
InstanceIds=instances,
DocumentName='CustomRunScript',
Comment='Starting init script from lambda prod',
CloudWatchOutputConfig={
'CloudWatchLogGroupName': 'some-group-name',
'CloudWatchOutputEnabled': True,
},
)
print('SSM response is: ', ssm_response)
When you send a command by using Run Command, you can specify where you want to send the command output. By default, Systems Manager returns only the first 2,500 characters of the command output. If you want to view the full details of the command output, you can specify an Amazon Simple Storage Service (Amazon S3) bucket. Or you can specify Amazon CloudWatch Logs. If you specify CloudWatch Logs, Run Command periodically sends all command output and error logs to CloudWatch Logs. You can monitor output logs in near real-time, search for specific phrases, values, or patterns, and create alarms based on the search.
Note you will need to provide proper IAM permissions for the Lambda to access the log group. Those permissions are listed in the reference below.
See Configuring Amazon CloudWatch Logs for Run Command

Send SNS Notification when EC2 instance is rebooted

I need help with one AWS Step. I know we can send an SNS Notification when the Instance is Stopped, terminated, starting and pending stages. But how do we send a notification when the EC2 instance is rebooted?
Thanks!
If a reboot is issued within the instance, then this will not be detected by AWS. It is just the operating system doing its own things.
If a reboot is issued through the AWS management console or an API call, the instance does not actually change state. From Instance Lifecycle - Amazon Elastic Compute Cloud:
Rebooting an instance is equivalent to rebooting an operating system. The instance remains on the same host computer and maintains its public DNS name, private IP address, and any data on its instance store volumes. It typically takes a few minutes for the reboot to complete, but the time it takes to reboot depends on the instance configuration.
Therefore, the only way to react to the reboot command being issued via the AWS console or API is to create an AWS CloudWatch Events rule that receives all Amazon EC2 events and then checks whether it is specifically for a RebootInstances command.
The rule would look like:
{
"source": [
"aws.ec2"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"ec2.amazonaws.com"
],
"eventName": [
"RebootInstances"
]
}
}
It can then trigger an Amazon SNS notification, which will include the instanceId.
However, the notification is not very pretty — it consists of a blob of JSON. If you want to send a nicer-looking message, see: amazon web services - Email notification through SNS and Lambda - Stack Overflow
Instead of monitoring Cloudtrail you could create a cron entry on instance that will execute #reboot (example here) and send sns notification using aws cli sns publish.
You can use the crontab with #reboot against a script to run.
$ crontab -e
#reboot $(python /home/ec2-user/sms.py)
for the python script sms.py at /home/ec2-user
sms.py (change the AWS region if required):
import boto3
import json
client = boto3.client('sns', region_name='us-west-2')
msg = 'Instance reboot!'
response = client.publish(
PhoneNumber='+1XXXXXXXXXX',
Message=msg)
print(response)
Make sure boto3 is installed on the instance - $ pip install boto3 --user

Is there a simple way to clone a glue job, but change the database connections?

I have a large number of clients who supply data in the same format, and need them loading into identical tables in different databases. I have set up a job for them in Glue, but now I have to do the same thing another 20 times
Is there any way I can take an existing job and copy it, but with changes to the S3 filepath and the JDBC connection?
I haven't been able to find much online regarding scripting in AWS Glue. Would this be achievable through the AWS command line interface?
The quickest way would be to use the aws cli.
aws glue get-job --job-name <value>
where value is the specific job that you are trying to replicate. You can then alter the s3 path and JDBC connection info in the JSON that the above command returns. Also, you'll need to give it a new unique name. Once you've done that, you can pass that in to:
aws glue create-job --cli-input-json <value>
where value is the updated JSON that you are trying to create a new job from.
See AWS command line reference for more info on the glue command line
use the command
aws glue create-job --generate-cli-skeleton
to generate the skeleton JSON
Use the below command to get the existing job's definition
aws glue get-job --job-name <value>
Copy the values from the output of existing job's definition into skeleton
Remove the newline character and pass it as input to below command
aws glue create-job --cli-input-json <framed_JSON>
Here is the complete reference for Create Job AWS CLI documentation
https://docs.aws.amazon.com/cli/latest/reference/glue/create-job.html
PS: don't change the order of the elements in JSON (generated in skeleton), only update the connection and name
--cli-input-json (string) Performs service operation based on the JSON string provided. The JSON string follows the format provided by --generate-cli-skeleton. If other arguments are provided on the command line, the CLI values will override the JSON-provided values. It is not possible to pass arbitrary binary values using a JSON-provided value as the string will be taken literally.
--generate-cli-skeleton (string) Prints a JSON skeleton to standard output without sending an API request. If provided with no value or the value input, prints a sample input JSON that can be used as an argument for --cli-input-json. If provided with the value output, it validates the command inputs and returns a sample output JSON for that command.
Thanks to the great answers here, you already know that the AWS CLI comes to the rescue.
Tip: if you don't want to install or update the AWS CLI, just use the AWS CloudShell!
I've tested the commands here using version:
$ aws --version
aws-cli/1.19.14 Python/3.8.5 Linux/5.4.0-65-generic botocore/1.20.14
If you want to create a new job from scratch, you'll want a template first, which you can get with:
aws glue create-job --generate-cli-skeleton > job_template.json
Then use your favourite editor (I like vim) to fill out the details in job_template.json (or whatever you call it).
But if DuckDuckGo or other engine sent you here, there's probably an existing job that you would like to clone and tweak. We'll call it "perfect_job" in this guide.
Let's get a list of all the jobs, just to check we're in the right place.
aws glue list-jobs --region us-east-1
The output shows us two jobs:
{
"JobNames": [
"perfect_job",
"sunshine"
]
}
View our job:
aws glue get-job --job-name perfect_job --region us-east-1
The JSON output looks right, let's put it in a file so we can edit it:
aws glue get-job --job-name perfect_job --region us-east-1 > perfect_job.json
Let's cp that to a new file, say  super_perfect_job.json. Now you can edit it to change the fields as desired. The first thing of course is to change the Name!
Two things to note:
Remove the outer level of the JSON, we need the value of Job not the Job identifier itself. If you look at job_template.json created above, you'll see that it must start with Name, so it's a small edit to match the format requirement.
There's no CreatedOn or LastModifiedOn in job_template.json either, so let's delete those lines too. Don't worry, if you forget to delete them, the creation will fail with a helpful message like 'Parameter validation failed: Unknown parameter in input: "LastModifiedOn"'.
Now we're ready to create the job! The following example will add Glue job "super_perfect_job" in the Cape Town region:
aws glue create-job --cli-input-json file://super_perfect_job.json --region af-south-1
But that didn't work:
An error occurred (InvalidInputException) when calling the CreateJob
operation: Please set only Allocated Capacity or Max Capacity.
I delete MaxCapacity and try again. Still not happy:
An error occurred (InvalidInputException) when calling the CreateJob
operation: Please do not set Allocated Capacity if using Worker Type
and Number of Workers.
Fine. I delete AllocatedCapacity and have another go. This time the output is:
{
    "Name": "super_perfect_job"
}
Which means, success! You can confirm by running list-jobs again. It's even more rewarding to open the AWS Console and see it pop up in the web UI.
We can't wait to run this job, so we'll use the CLI as well, and we'll pass three additional parameters: --fruit, --vegetable and --nut which our script expects. But -- would confuse the AWS CLI so let's store these in a file called args.json containing:
{
  "--fruit": "tomato",
  "--vegetable": "cucumber",
  "--nut": "almond"
}
And call our job like so:
aws glue start-job-run --job-name super_perfect_job --arguments file://args.json --region af-south-1
Or like this:
aws glue start-job-run --job-name super_perfect_job --arguments '{"--fruit": "tomato","--vegetable": "cucumber"}'
And you can view the status of job runs with:
aws glue get-job-runs --job-name super_perfect_job --region us-east-1
As you can see, the AWS Glue API accessed by the AWS CLI is pretty powerful, being not only convenient, but allowing automation in Continuous Integration (CI) servers like Jenkins, for example. Run aws glue help for more commands and quick help or see the online documentation for more details.
For creating or managing permanent infrastructure, it's preferable to use Infrastructure as Code tools, such as CloudFormation or Terraform.

Is there a direct way- a cft to list all lambda functions for a particular region?

I want to have a cloud formation template to list all lambda functions for a particular region. I don't need to write a lambda code using list-function and call it inside my CFT.
I tried incorporating CLI command inside CFT but it didn't work
There is no way to directly add a aws cli command in a cloudformation template. Either you will have to create a EC2 instance and then run the CLI command in the user data or create a lambda backed custom resource to do it.
Both will complicate the simple CLI command.
aws lambda list-functions --region eu-west-1
CFN is just an orchestration tool. It cannot compute on itself.
Instead we can use a simple lambda python script and invoke the same in the CFT
import boto3
#Create an lambda client
client = boto3.client(
"lambda"
)
response = client.list_functions(
MasterRegion='string',
FunctionVersion='ALL',
Marker='string',
MaxItems=123
)
print(response)

Using AWS SNS when ec2 instance is deployed in us-west-1

I have a quick question about usage of AWS SNS.
I have deployed an EC2 (t2.micro, Linux) instance in us-west-1 (N.California). I have written a python script using boto3 to send a simple text message to my phone. Later I discovered, there is no SNS service for instances deployed out of us-east-1 (N.Virginia). Till this point it made sense, because I see this below error when i execute my python script, as the region is defined as "us-west-1" in aws configure (AWS cli) and also in my python script.
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameter) when calling the Publish operation: Invalid parameter: PhoneNumber Reason:
But to test, when I changed the "region" in aws conifgure and in my python script to "us-east-1", my script pushed a text message to my phone. Isn't it weird? Can anyone please explain why this is working just by changing region in AWS cli and in my python script, though my instance is still in us-west-1 and I dont see "Publish text message" option on SNS dashboard on N.california region?
Is redefining the aws cli with us-east-1 similar to deploying a new instance altogether in us-east-1? I dont think so. Correct me if I am wrong. Or is it like having an instance in us-west-1, but just using SNS service from us-east-1? Please shed some light.
Here is my python script, if anyone need to look at it (Its a simple snippet).
import boto3
def send_message():
# Create an SNS client
client = boto3.client("sns", aws_access_key_id="XXXX", aws_secret_access_key="XXXX", region_name="us-east-1")
# Send your sms message.
client.publish(PhoneNumber="XXXX",Message="Hello World!")
if __name__ == '__main__':
send_message()
Is redefining the aws cli with us-east-1 similar to deploying a new
instance altogether in us-east-1?
No, it isn't like that at all.
Or is it like having an instance in us-west-1, but just using SNS
service from us-east-1?
Yes, that's all you are doing. You can connect to any AWS regions' API from anywhere on the Internet. It doesn't matter that it is running on an EC2 instance in a specific region, it only matters what region you tell the SDK/CLI to use.
You could run the same code on your local computer. Obviously your local computer is not running on AWS so you would have to tell the code which AWS region to send the API calls to. What you are doing is the same thing.
Code running on an EC2 server is not limited into using the AWS API in the same region that the EC2 server is in.
Did you try creating a topic before publishing to it? You should try create a topic and then publish to that topic.