AWS Lambda scheduled event source via cloudformation - amazon-web-services

I already have my lambda / roles defined in cloudformation and would love to also use it to add a scheduled eventsources ... are there any docs or examples around ?

Use Aws::Event::Rule with a ScheduleExpression and a AWS::Lambda::Permission
// rule to periodically call the lambda
"TagWatcherRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"ScheduleExpression": "rate(10 minutes)",
"Targets": [
{
"Id": "TagWatcherScheduler",
"Arn": {
"Fn::GetAtt": [
"TagWatcherFunction",
"Arn"
]
}
}
]
}
},
// role may call the lambda
"InvokeLambdaPermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Fn::GetAtt": [
"TagWatcherFunction",
"Arn"
]
},
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceArn": {
"Fn::GetAtt": [
"TagWatcherRule",
"Arn"
]
}
}
}

I solved same problem.
"RoleForLambdaStopEC2Instances" : {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Policies": [
{
"PolicyName": "LambdaStopEC2InstancesPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"ec2:StopInstances"
],
"Resource": [
"arn:aws:logs:*:*:*",
"arn:aws:ec2:*"
]
}
]
}
}
],
"Path": "/"
}
},
"LambdaStopEC2Instances": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": "XXXXXXXXXXXXXXXXX",
"S3Key": "XXXXXXXXXXXXXXXXXX"
},
"Handler": "stopEC2Instances.handler",
"Role": { "Fn::GetAtt" : ["RoleForLambdaStopEC2Instances", "Arn"] },
"Runtime": "nodejs4.3",
"Timeout": "5"
}
},
"StopEC2InstancesRule": {
"Type" : "AWS::Events::Rule",
"Properties" : {
"Name" : "StopEC2Instances",
"ScheduleExpression" : "cron(0 13 ? * MON-FRI *)",
"State": "ENABLED",
"Targets": [{
"Arn": { "Fn::GetAtt": ["LambdaStopEC2Instances", "Arn"] },
"Id": "stopEC2Instances"
}]
}
},
"LambdaInvokePermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName" : { "Fn::GetAtt" : ["LambdaStopEC2Instances", "Arn"] },
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceAccount": { "Ref" : "AWS::AccountId" },
"SourceArn": { "Fn::GetAtt": ["StopEC2InstancesRule","Arn"] }
}
}

Unfortunately, configuring scheduled event sources for lambda functions is currently not supported by CloudFormation. You will need to deploy your lambda using CloudFormation and then manually configure your scheduled events.
CloudFormation does support an AWS::Lambda::EventSourceMapping resource type. However, this resource is limited configuring Kinesis or DynamoDB streams, so this is likely not helpful to you.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html
**Update - as of April 2016, this is now supported using CloudWatch Events - https://aws.amazon.com/about-aws/whats-new/2016/04/amazon-cloudwatch-events-now-supported-in-aws-cloudformation-templates/

The YAML Version
ScheduledRule:
Type: AWS::Events::Rule
Properties:
Description: "ScheduledRule"
ScheduleExpression: "rate(10 minutes)"
State: "ENABLED"
Targets:
-
Arn:
Fn::GetAtt:
- "LambdaFunction"
- "Arn"
Id: "TargetFunctionV1"
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Ref: "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- "ScheduledRule"
- "Arn"

As of this week (18 April 2016) it is now possible to add a scheduled CloudWatch event rule that will trigger your Lambda function.
The AWS::Event::Rule has a ScheduleExpression field for the cron-style schedule and a Targets array which can accept a Lambda function ARN.

AWS supports periodic run through sourcedetails.
EventSource: "aws.config"
MaximumExecutionFrequency: Twelve_Hours
MessageType: "ScheduledNotification"

If you use function name as
"FunctionName": {
"Fn::GetAtt": [
"TagWatcherFunction",
"Arn"
]
}
and you not specify the function then it'll throw you
"Template is not valid: Template error: instance of Fn::GetAtt references undefined resource TagWatcherFunction"
So instead of function name you can directly specify the "lambda ARN".
you can see example below
"TagWatcherRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"ScheduleExpression": "rate(10 minutes)",
"Targets": [
{
"Id": "TagWatcherScheduler",
"Arn": {
"Fn::GetAtt": [
"TagWatcherFunction",
"Arn"
]
}
}
]
}
},
// role may call the lambda
"InvokeLambdaPermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": "arn:aws:lambda:<region>:<awsid>:function:<lambd name>",
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceArn": {
"Fn::GetAtt": [
"TagWatcherRule",
"Arn"
]
}
}
}

Related

Need help on CloudFormation template and AWS lambda for pulling events from SQS to S3 via lambda

I am new to AWS CloudFormation, and I am trying to capture events from SQS Queue and place them in S3 bucket via AWS lambda. Flow of events is
SNS --> SQS <-- Lambda ---> S3 bucket.
I am trying to achieve above flow using cloudFormation template.I am getting below error message after deploying my CloudFormation template. Any help you can provide would be greatly appreciated. Thank you
11:51:56 2022-01-13 17:51:47,930 - INFO - ...
11:52:53 2022-01-13 17:52:48,511 - ERROR - Stack myDemoApp shows a rollback status ROLLBACK_IN_PROGRESS.
11:52:53 2022-01-13 17:52:48,674 - INFO - The following root cause failure event was found in the myDemoApp stack for resource EventStreamLambda:
11:52:53 2022-01-13 17:52:48,674 - INFO - Resource handler returned message: "Error occurred while GetObject. S3 Error Code: NoSuchKey.
S3 Error Message: The specified key does not exist. (Service: Lambda, Status Code: 400, Request
ID: 5f2f9882-a863-4a58-90bd-7e0d0dfdf4d5, Extended Request ID: null)" (RequestToken: 0a95acb4-a677-0a2d-d0bc-8b7487a858ad, HandlerErrorCode: InvalidRequest)
11:52:53 2022-01-13 17:52:48,674 - INFO - ..
MY lambda function is :
import json
import logging
import boto3
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logging.basicConfig(level=logging.INFO,
format='%(asctime)s: %(levelname)s: %(message)s')
def lambda_handler(event, context):
logger.info(f"lambda_handler -- event: {json.dumps(event)}")
s3_bucket = boto3.resource("3")
event_message = json.loads(event["Records"][0]["body"])
s3_bucket.put_object(Bucket="S3DeployBucket", key="data.json", Body=json.dumps(event_message))
My complete CloudFormation template is :
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "myDemoApp Resource Stack",
"Mappings": {
},
"Parameters": {
"S3DeployBucket": {
"Default": "myDemoApp-deploy-bucket",
"Description": "Bucket for deployment configs and artifacts for myDemoApp",
"Type": "String"
},
"EnvName": {
"Description": "Platform environment name for myDemoApp",
"Type": "String"
},
"AuditRecordKeyArn": {
"Description": "ARN for audit record key encryption for myDemoApp",
"Type": "String"
},
"ParentVPCStack": {
"Description": "The name of the stack containing the parent VPC for myDemoApp",
"Type": "String"
},
"StackVersion": {
"Description": "The version of this stack of myDemoApp",
"Type": "String"
},
"EventLogFolderName": {
"Type": "String",
"Description": "folder name for the logs for the event stream of myDemoApp",
"Default": "event_log_stream"
},
"EventLogPartitionKeys": {
"Type": "String",
"Description": "The partition keys that audit logs will write to S3. Use Hive-style naming conventions for automatic Athena/Glue comprehension.",
"Default": "year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}"
},
"AppEventSNSTopicArn": {
"Description": "Events SNS Topic of myDemoApp",
"Type": "String"
},
"ReportingEventsRetentionDays": {
"Default": "2192",
"Description": "The number of days to retain a record used for reporting.",
"Type": "String"
}
},
"Resources": {
"AppEventSQSQueue": {
"Type": "AWS::SQS::Queue"
},
"AppEventSnsSubscription": {
"Type": "AWS::SNS::Subscription",
"Properties": {
"TopicArn": {
"Ref": "AppEventSNSTopicArn"
},
"Endpoint": {
"Fn::GetAtt": [
"AppEventSQSQueue",
"Arn"
]
},
"Protocol": "sqs"
}
},
"S3DeployBucket": {
"Type": "AWS::S3::Bucket",
"DeletionPolicy": "Retain",
"UpdateReplacePolicy": "Retain",
"Properties": {
"BucketEncryption": {
"ServerSideEncryptionConfiguration": [
{
"ServerSideEncryptionByDefault": {
"KMSMasterKeyID": {
"Ref": "AuditRecordKeyArn"
},
"SSEAlgorithm": "aws:kms"
}
}
]
},
"VersioningConfiguration": {
"Status": "Enabled"
},
"LifecycleConfiguration": {
"Rules": [
{
"ExpirationInDays": {
"Ref": "ReportingEventsRetentionDays"
},
"Status": "Enabled"
}
]
}
}
},
"EventStreamLogGroup": {
"Type": "AWS::Logs::LogGroup"
},
"EventLogStream": {
"Type": "AWS::Logs::LogStream",
"Properties": {
"LogGroupName": {
"Ref": "EventStreamLogGroup"
}
}
},
"EventStreamSubscriptionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Policies": [
{
"PolicyName": "SNSSQSAccessPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": {
"Action": [
"sqs:*"
],
"Effect": "Allow",
"Resource": "*"
}
}
}
]
}
},
"EventDeliveryRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sqs.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": {
"Ref": "AWS::AccountId"
}
}
}
}
]
}
}
},
"EventSqsQueuePolicy": {
"Type": "AWS::SQS::QueuePolicy",
"Properties": {
"PolicyDocument": {
"Version": "2012-10-17",
"Id": "SqsQueuePolicy",
"Statement": [
{
"Sid": "Allow-SNS-SendMessage",
"Effect": "Allow",
"Principal": "*",
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage"
],
"Resource": {
"Fn::GetAtt": [
"EventStreamLambda",
"Arn"
]
},
"Condition": {
"ArnEquals": {
"aws:SourceArn": {
"Ref": "EventSNSTopicArn"
}
}
}
}
]
},
"Queues": [
{
"Ref": "EventSNSTopicArn"
}
]
}
},
"EventDeliveryPolicy": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": "sqs_delivery_policy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
{
"Fn::GetAtt": [
"S3DeployBucket",
"Arn"
]
},
{
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"S3DeployBucket",
"Arn"
]
},
"/*"
]
]
}
]
},
{
"Effect": "Allow",
"Action": [
"logs:PutLogEvents"
],
"Resource": {
"Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${EventStreamLogGroup}:log-stream:${EventLogStreamLogStream}"
}
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": [
{
"Ref": "AuditRecordKeyArn"
}
],
"Condition": {
"StringEquals": {
"kms:ViaService": {
"Fn::Join": [
"",
[
"s3.",
{
"Ref": "AWS::Region"
},
".amazonaws.com"
]
]
}
},
"StringLike": {
"kms:EncryptionContext:aws:s3:arn": {
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"S3DeployBucket",
"Arn"
]
},
"/*"
]
]
}
}
}
}
]
},
"Roles": [
{
"Ref": "EventDeliveryRole"
}
]
}
},
"EventStreamLambda": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "lambda_function.lambda_handler",
"MemorySize": 128,
"Runtime": "python3.8",
"Timeout": 30,
"FunctionName": "sqs_s3_pipeline_job",
"Role": {
"Fn::GetAtt": [
"SQSLambdaExecutionRole",
"Arn"
]
},
"Code": {
"S3Bucket": {
"Ref": "S3DeployBucket"
},
"S3Key": {
"Ref": "S3DeployBucket"
}
},
"TracingConfig": {
"Mode": "Active"
}
}
},
"SQSLambdaExecutionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Policies": [
{
"PolicyName": "StreamLambdaLogs",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": "arn:aws:logs:*:*:*"
}
]
}
},
{
"PolicyName": "SQSLambdaPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:GetQueueAttributes",
"sqs:ChangeMessageVisibility"
],
"Resource":"*"
}
]
}
}
]
}
}
},
"Outputs": {
"VpcSubnet3ExportKey": {
"Value": {
"Fn::Sub": "${ParentVPCStack}-privateSubnet3"
}
}
}
}
SubscriptionRoleArn is only for kinesis:
This property applies only to Amazon Kinesis Data Firehose delivery stream subscriptions.

Cloudformation Lambda function, environment variables not available, input is not available

Not sure where I am doing wrong. Below is my lambda code and cloudformation template.
Below is my lambda function:
import os
import boto3
def quickstart_lambda_handler(event, context):
print("event={}".format(event)) # nothing shows as k1:v1
print(os.environ) # nothing show in output as var1
print("environment variable var1: " + os.environ['var1'])
json_str = "{\"a1\":\"b1\"}"
bucket_name = "my-s3-bucket"
file_name = "qs.json"
s3_path = "cf-quickstart/" + file_name
s3 = boto3.resource("s3")
s3.Bucket(bucket_name).put_object(Key=s3_path, Body=json_str)
print("object stored in s3")
return s3_path
My cloud formation template is as below for lambda function and associated rule.
"QSLERole": {
"Description" : "Quick Start Lambda Execution Role",
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName" : "QSLERole",
"AssumeRolePolicyDocument": {
"Version" : "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [ "lambda.amazonaws.com" ]
},
"Action": [ "sts:AssumeRole" ]
}]
},
"Path": "/",
"Policies": [
{
"PolicyName": "root",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["logs:*"],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "*"
}
]
}
}]
}
},
"QSLambdaF": {
"Description" : "Quick Start Lambda Function",
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": "my-s3-bucket",
"S3Key": "generate-quickstartjson.zip"
},
"Handler": "generate-quickstartjson.quickstart_lambda_handler",
"Runtime": "python2.7",
"Timeout": "30",
"Environment" : {
"Variables" : {
"var1" : "var1 value"
}
},
"Role": {
"Fn::GetAtt": [
"QSLERole",
"Arn"
]
}
}
},
"ScheduledRule": {
"Description" :"Cloud Watch Quick Start Event Rule",
"DependsOn" : [
"QSLambdaF","EC2Instance"],
"Type": "AWS::Events::Rule",
"Properties" : {
"Description" : "CloudWatch Event for generating output",
"ScheduleExpression" : "rate(1 minute)",
"State": "ENABLED",
"Targets" : [
{
"Arn": { "Fn::GetAtt": [
"QSLambdaF", "Arn"] },
"Detail" : {
"test" : "value"
},
"Input": "{\"k1\":\"v1\"}",
"Id" : "QSScheduledRuleV1"
}
]
}
},
"PEILambda": {
"Description" : "Permission For Events To Invoke Lambda Function",
"Type": "AWS::Lambda::Permission",
"Properties" : {
"FunctionName": { "Ref": "QSLambdaF"},
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceArn": { "Fn::GetAtt": [
"ScheduledRule", "Arn"] }
}
},
--- ec2 instance creation below with security group
Niether environment variables are available nor Input fields are available in lambda lambda.
"Input": "{\"k1\":\"v1\"}",
Please let me know where I am wrong. Thanks for the help.
works well as is. I was using wrong template file :))

AWS Deploying environment and create environments for dev and prod

Greeting all,
I'm looking for a way to deploy my application which contains:
API Gateway
DynamoDB
Lambda Functions
An S3 bucket
I looked at CloudFormation and CodeDeploy but I'm unsure how to proceed without EC2...
All the information I find is for EC2, I haven't found any information regarding deploying the app above...
The goal is to have a deployment script that deploys app to an environment automatically with technology from AWS. (Basically duplicating my environment)
Any help would greatly be appreciated.
EDIT: I need to be able to export from one AWS account then import onto another AWS account.
Cheers!
In order to deploy your CloudFormation stack into a "different" environment, you have to parameterize your CloudFormation stack name and resource names. (You don't have to parameterize the AWS::Serverless::Function function in this example because CloudFormation automatically creates a function name if no function name is specified, but for most other resources it's necessary)
Example CloudFormation template cfn.yml using the Serverless Application Model (SAM):
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Deploys a simple AWS Lambda using different environments.
Parameters:
Env:
Type: String
Description: The environment you're deploying to.
Resources:
ServerlessFunction:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: nodejs12.x
CodeUri: ./
Policies:
- AWSLambdaBasicExecutionRole
MyBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub 'my-bucket-name-${Env}'
You can add further resources like a DynamoDB table. The API Gateway is automatically created if you're using SAM and provide an Events section in your AWS::Serverless::Function resource. See also this SAM example code from the serverless-app-examples repository.
Example deploy.sh script:
#!/usr/bin/env bash
LAMBDA_BUCKET="Your-S3-Bucket-Name"
# change this ENV variable depending on the environment you want to deploy
ENV="prd"
STACK_NAME="aws-lambda-cf-environments-${ENV}"
# now package the CloudFormation template which automatically uploads the Lambda function artifacts to S3 -> generated a "packaged" CloudFormation template cfn.packaged.yml
aws cloudformation package --template-file cfn.yml --s3-bucket ${LAMBDA_BUCKET} --output-template-file cfn.packaged.yml
# ... and deploy the packaged CloudFormation template
aws cloudformation deploy --template-file cfn.packaged.yml --stack-name ${STACK_NAME} --capabilities CAPABILITY_IAM --parameter-overrides Env=${ENV}
See the full example code here. Just deploy the script using ./deploy.sh and change the ENV variable.
Based JSON examples.
Lambda function AWS::Lambda::Function
This example creates a Lambda function and an IAM Role attached to it.
Language: NodeJs.
"LambdaRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Policies": [
{
"PolicyName": "LambdaSnsNotification",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowSnsActions",
"Effect": "Allow",
"Action": [
"sns:Publish",
"sns:Subscribe",
"sns:Unsubscribe",
"sns:DeleteTopic",
"sns:CreateTopic"
],
"Resource": "*"
}
]
}
}
]
}
},
"LambdaFunctionMessageSNSTopic": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Description": "Send message to a specific topic that will deliver MSG to a receiver.",
"Handler": "index.handler",
"MemorySize": 128,
"Role": {
"Fn::GetAtt": [
"LambdaRole",
"Arn"
]
},
"Runtime": "nodejs6.10",
"Timeout": 60,
"Environment": {
"Variables": {
"sns_topic_arn": ""
}
},
"Code": {
"ZipFile": {
"Fn::Join": [
"\n",
[
"var AWS = require('aws-sdk');",
"};"
]
]
}
}
}
}
API Gateway AWS::ApiGateway::RestApi
This example creates Role, RestAPI, Usageplan, Keys and permission to execute lambda from a Request method.
"MSGGatewayRestApi": {
"Type": "AWS::ApiGateway::RestApi",
"Properties": {
"Name": "MSG RestApi",
"Description": "API used for sending MSG",
"FailOnWarnings": true
}
},
"MSGGatewayRestApiUsagePlan": {
"Type": "AWS::ApiGateway::UsagePlan",
"Properties": {
"ApiStages": [
{
"ApiId": {
"Ref": "MSGGatewayRestApi"
},
"Stage": {
"Ref": "MSGGatewayRestApiStage"
}
}
],
"Description": "Usage plan for stage v1",
"Quota": {
"Limit": 5000,
"Period": "MONTH"
},
"Throttle": {
"BurstLimit": 200,
"RateLimit": 100
},
"UsagePlanName": "Usage_plan_for_stage_v1"
}
},
"RestApiUsagePlanKey": {
"Type": "AWS::ApiGateway::UsagePlanKey",
"Properties": {
"KeyId": {
"Ref": "MSGApiKey"
},
"KeyType": "API_KEY",
"UsagePlanId": {
"Ref": "MSGGatewayRestApiUsagePlan"
}
}
},
"MSGApiKey": {
"Type": "AWS::ApiGateway::ApiKey",
"Properties": {
"Name": "MSGApiKey",
"Description": "CloudFormation API Key v1",
"Enabled": "true",
"StageKeys": [
{
"RestApiId": {
"Ref": "MSGGatewayRestApi"
},
"StageName": {
"Ref": "MSGGatewayRestApiStage"
}
}
]
}
},
"MSGGatewayRestApiStage": {
"DependsOn": [
"ApiGatewayAccount"
],
"Type": "AWS::ApiGateway::Stage",
"Properties": {
"DeploymentId": {
"Ref": "RestAPIDeployment"
},
"MethodSettings": [
{
"DataTraceEnabled": true,
"HttpMethod": "*",
"LoggingLevel": "INFO",
"ResourcePath": "/*"
}
],
"RestApiId": {
"Ref": "MSGGatewayRestApi"
},
"StageName": "v1"
}
},
"ApiGatewayCloudWatchLogsRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"apigateway.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Policies": [
{
"PolicyName": "ApiGatewayLogsPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents",
"logs:GetLogEvents",
"logs:FilterLogEvents"
],
"Resource": "*"
}
]
}
}
]
}
},
"ApiGatewayAccount": {
"Type": "AWS::ApiGateway::Account",
"Properties": {
"CloudWatchRoleArn": {
"Fn::GetAtt": [
"ApiGatewayCloudWatchLogsRole",
"Arn"
]
}
}
},
"RestAPIDeployment": {
"Type": "AWS::ApiGateway::Deployment",
"DependsOn": [
"MSGGatewayRequest"
],
"Properties": {
"RestApiId": {
"Ref": "MSGGatewayRestApi"
},
"StageName": "DummyStage"
}
},
"ApiGatewayMSGResource": {
"Type": "AWS::ApiGateway::Resource",
"Properties": {
"RestApiId": {
"Ref": "MSGGatewayRestApi"
},
"ParentId": {
"Fn::GetAtt": [
"MSGGatewayRestApi",
"RootResourceId"
]
},
"PathPart": "delivermessage"
}
},
"MSGGatewayRequest": {
"DependsOn": "LambdaPermission",
"Type": "AWS::ApiGateway::Method",
"Properties": {
"ApiKeyRequired": true,
"AuthorizationType": "NONE",
"HttpMethod": "POST",
"Integration": {
"Type": "AWS",
"IntegrationHttpMethod": "POST",
"Uri": {
"Fn::Join": [
"",
[
"arn:aws:apigateway:",
{
"Ref": "AWS::Region"
},
":lambda:path/2015-03-31/functions/",
{
"Fn::GetAtt": [
"LambdaFunctionMessageSNSTopic",
"Arn"
]
},
"/invocations"
]
]
},
"IntegrationResponses": [
{
"StatusCode": 200
},
{
"SelectionPattern": "500.*",
"StatusCode": 500
},
{
"SelectionPattern": "412.*",
"StatusCode": 412
}
],
"RequestTemplates": {
"application/json": ""
}
},
"RequestParameters": {
},
"ResourceId": {
"Ref": "ApiGatewayMSGResource"
},
"RestApiId": {
"Ref": "MSGGatewayRestApi"
},
"MethodResponses": [
{
"StatusCode": 200
},
{
"StatusCode": 500
},
{
"StatusCode": 412
}
]
}
},
"LambdaPermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"Action": "lambda:invokeFunction",
"FunctionName": {
"Fn::GetAtt": [
"LambdaFunctionMessageSNSTopic",
"Arn"
]
},
"Principal": "apigateway.amazonaws.com",
"SourceArn": {
"Fn::Join": [
"",
[
"arn:aws:execute-api:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":",
{
"Ref": "MSGGatewayRestApi"
},
"/*"
]
]
}
}
}
DynamoDB AWS::DynamoDB::Table
This example creates a DynamoDB table MyCrossConfig and an alarms for it.
"TableMyCrossConfig": {
"Type": "AWS::DynamoDB::Table",
"Properties": {
"TableName": "MyCrossConfig",
"AttributeDefinitions": [
{
"AttributeName": "id",
"AttributeType": "S"
}
],
"KeySchema": [
{
"AttributeName": "id",
"KeyType": "HASH"
}
],
"ProvisionedThroughput": {
"ReadCapacityUnits": "5",
"WriteCapacityUnits": "5"
}
}
},
"alarmTargetTrackingtableMyCrossConfigProvisionedCapacityLowdfcae8d90ee2487a8e59c7bc0f9f6bd9": {
"Type": "AWS::CloudWatch::Alarm",
"Properties": {
"ActionsEnabled": "true",
"AlarmDescription": {
"Fn::Join": [
"",
[
"DO NOT EDIT OR DELETE. For TargetTrackingScaling policy arn:aws:autoscaling:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":scalingPolicy:7558858e-b58c-455c-be34-6de387a0c6d1:resource/dynamodb/table/MyCrossConfig:policyName/DynamoDBReadCapacityUtilization:table/MyCrossConfig."
]
]
},
"ComparisonOperator": "LessThanThreshold",
"EvaluationPeriods": "3",
"MetricName": "ProvisionedReadCapacityUnits",
"Namespace": "AWS/DynamoDB",
"Period": "300",
"Statistic": "Average",
"Threshold": "5.0",
"AlarmActions": [
{
"Fn::Join": [
"",
[
"arn:aws:autoscaling:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":scalingPolicy:7558858e-b58c-455c-be34-6de387a0c6d1:resource/dynamodb/table/MyCrossConfig:policyName/DynamoDBReadCapacityUtilization:table/MyCrossConfig"
]
]
}
],
"Dimensions": [
{
"Name": "TableName",
"Value": "MyCrossConfig"
}
]
}
}
s3 bucket AWS::S3::Bucket
This example creates a Bucket with name configbucket- + AWS::AccountId
"ConfigBucket": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": {
"Fn::Join": [
"",
[
"configbucket-",
{
"Ref": "AWS::AccountId"
}
]
]
}
},
"DeletionPolicy": "Delete"
}
Now you need to put altogether, make the reference in the template, Etc.
Hope it helps!
My guess would be that you could use CloudFormation for such an app, but I'm also unfamiliar.
What I have had success with is writing small scripts which leverage the awscli utility to accomplish this. Additionally, you'll need a strategy for how you setup a new environment.
Typically, what I have done is to use a different suffix on DynamoDB tables and S3 buckets to represent different environments. Lambda + API Gateway have the idea of different versions baked in, so you can support different environments there as well.
For really small projects, I have even setup my Dynamo schema to support many environments within a single table. This is nice for pet or small projects because it's cheaper.
Built my own SDK for deployments, it's in the making...
https://github.com/LucLaverdure/aws-sdk
You will need to use the following shell scripts within the containers:
export.sh
import.sh
Requirements:
AWS CLI
Python
pip
npm
jq

cloudwatchevent_rule default to latest version of lambda function

I am trying to automate the creation of a lambda function and cloudwatch rule for it. However it seems that the cloudwatchevent_rule ansible task requires a version id to attach itself to my lambda function. This is causing an error:
No target to arn:aws:lambda:us-east-
1:MYACCOUNTID:function:MYFUNCTIONNAME could be found on the rule
MYFUNCTIONNAME.
How can I change this so that the cloudwatch rule will always attach itself to the latest version of my lambda function:
- name: create cloudwatch rule
cloudwatchevent_rule:
name: 'name_for_rule'
region: "{{region}}"
description: 'trigger on new instance creation'
state: present
event_pattern: |-
{
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"ec2.amazonaws.com"
],
"eventName": [
"RunInstances"
]
}
}
targets:
- id: "{{ lambda.configuration.version }}"
arn: "{{ lambda.configuration.function_arn }}"
I've configured a Lambda Function with CloudWatch rule triggering it. The following SAM Template also contains permission, policy and roles I require. Please ignore those if not required.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Transform": "AWS::Serverless-2016-10-31",
"Description": "AWS SAM template configuring lambda functions written in test package.",
"Resources": {
"OrchestratorTestLambdaFunction": {
"DependsOn": [
"LambdaPolicy"
],
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "com.test.TestClass::orchestrateTestLambda",
"FunctionName": "OrchestratorTestLambda",
"Runtime": "java8",
"MemorySize": 256,
"Timeout": 60,
"Code": {
"S3Bucket": "BATS::SAM::CodeS3Bucket",
"S3Key": "BATS::SAM::CodeS3Key"
},
"Role": {
"Fn::GetAtt": [
"LambdaRole",
"Arn"
]
},
"Description": "Lambda reads from SQS provided in the cloud watch."
}
},
"LambdaRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": "LambdaRole",
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
}
},
"LambdaPolicy": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": "lambda_policy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"sqs:DeleteMessage",
"sqs:ReceiveMessage"
],
"Resource": [
{
"Fn::Sub": "arn:aws:sqs:eu-west-1:${AWS::AccountId}:TestUpdates"
}
]
},
{
"Sid": "",
"Action": [
"lambda:InvokeAsync"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Sid": "",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
}
]
},
"Roles": [
{
"Ref": "LambdaRole"
}
]
}
},
"PermissionForEventsToInvokeLambda": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Ref": "OrchestratorTestLambdaFunction"
},
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceArn": {
"Fn::GetAtt": [
"TestRule",
"Arn"
]
}
}
},
"TestRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"Name": "TestRule",
"Description": "Rule to Trigger OrchestratorTestLambdaFunction",
"ScheduleExpression": "rate(1 minute)",
"State": "ENABLED",
"Targets": [
{
"Arn": {
"Fn::GetAtt": [
"OrchestratorTestLambdaFunction",
"Arn"
]
},
"Id": "TestRuleV1",
"Input": {
"Fn::Sub": "{\"queueUrl\":\"https://sqs.eu-west-1.amazonaws.com/${AWS::AccountId}/TestUpdates\"}"
}
}
]
}
}
},
"Outputs": {
"StackArn": {
"Value": {
"Ref": "AWS::StackId"
},
"Description": "Use this as the stack_arn in your cloud_formation_deployment_stack override."
}
}
}
I've noticied that function_arn registered from lambda ansible module output is not consistent.
Some times is
"function_arn": "arn:aws:lambda:zone:account:function:name"
other time is:
"function_arn": "arn:aws:lambda:zone:account:function:name:version"
So I've construct the arn appending always the $LATEST version:
- cloudwatchevent_rule:
profile: "{{ aws_profile }}"
name: StartStop
schedule_expression: cron(* * * * ? *)
description: trigger my lambda
targets:
- id: StartStop
arn: "arn:aws:lambda:{{aws_zone}}:{{aws_account_id}}:function:{{lambdadeploy.configuration.function_name}}:$LATEST"

SNS topic not triggering Lambda

I am attempting to set up a email-sending lambda function that is triggered by an SNS topic in cloudformation, but for some reason it is not working. I went in and checked all of the dependencies/permissions after the lambda & sns went up and everything seems to be in order, but when I publish to the topic nothing happens. When I manually test the lambda in the Lambda console, it works perfectly.
Cloudformation
"Resources": {
"CloudformationEventHandlerLambdaExecutionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"Path": "/",
"Policies": [
{
"PolicyName": "CloudformationTrigger",
"PolicyDocument": {
"Statement": [
{
"Effect": "Allow",
"Action": [
"ses:*"
],
"Resource": [
"arn:aws:ses:*"
]
}
]
}
}
],
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": [
"sts:AssumeRole"
],
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
}
}
]
}
}
},
"CloudformationEventHandlerLambdaFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "lambda_function.lambda_handler",
"Role": {
"Fn::GetAtt": [
"CloudformationEventHandlerLambdaExecutionRole",
"Arn"
]
},
"Code": {
"S3Bucket": {
"Ref": "Bucket"
},
"S3Key": "CloudformationEventHandler.zip"
},
"Runtime": "python2.7",
"Timeout": "30"
},
"DependsOn": [
"CloudformationEventHandlerLambdaExecutionRole"
]
},
"CloudformationEventHandlerLambdaInvokePermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"Action": "lambda:InvokeFunction",
"SourceAccount": {
"Ref": "AWS::AccountId"
},
"Principal": "sns.amazonaws.com",
"SourceArn": {
"Ref": "CloudformationTopic"
},
"FunctionName": {
"Fn::GetAtt": [
"CloudformationEventHandlerLambdaFunction",
"Arn"
]
}
}
},
"CloudformationTopic": {
"Type": "AWS::SNS::Topic",
"Properties": {
"DisplayName": "CloudformationIngestTopic",
"Subscription": [
{
"Endpoint": {
"Fn::GetAtt": [
"CloudformationEventHandlerLambdaFunction",
"Arn"
]
},
"Protocol": "lambda"
}
]
},
"DependsOn": [ "CloudformationEventHandlerLambdaFunction" ]
}
}
Python SES Lambda
import boto3
client = boto3.client('ses')
def lambda_handler(event, context):
message = """
Event:
{}
Context:
{}
""".format(event, context)
response = client.send_email(
Source='***censored***',
Destination={ 'ToAddresses': [ ***censored***' ] },
Message={
'Subject': {
'Data': 'CFMTest'
},
'Body': {
'Text': {
'Data': message
}
}
}
)
The SourceAccount for the AWS::Lambda::Permission resource type is only meant to be used with Cloudwatch logs, CloudWatch rules, S3 and SES.
After removing this field from the CloudformationEventHandlerLambdaInvokePermission resource on your template, I am able to invoke the lambda function by publishing to the SNS topic.
Refer to this documentation for more information regarding lambda permissions