Within AWS I've created a CloudTrail which is then filtered by an Eventbridge Rule to only look for certain events within CloudTrail that correspond to resources being created on AWS. Reason being is that I've created some code for Lambda that will automatically tag resources dependent on events passed to it from Eventbridge. When I manually connect eventbridge rule & lambda together it all works fine. However, when I deploy my stack using Cloudformation my Lambda doesn't show Eventbridge as an eventsource/trigger for the function and I don't know why. Below is my Cloudformation template alongside what is shown on AWS Lambda vs what I expect to be seen.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
# ---------------------------------------------------------------------------- #
# Input parameters #
# ---------------------------------------------------------------------------- #
Parameters:
ProjectName:
Type: String
Default: 'AutoTagger'
Description: ""
# ---------------------------------------------------------------------------- #
# Resources #
# ---------------------------------------------------------------------------- #
Resources:
AutoTaggerLambda:
Type: AWS::Serverless::Function
Name: auto-tagger-lambda
Properties:
CodeUri: release/archive.zip
Handler: auto-tagger/main.lambda_handler
Runtime: python3.9
Policies: [AWSLambda_FullAccess]
MemorySize: 128
Timeout: 30
Tags:
- Key: "project_name"
Value: !Ref ProjectName
TagEvents:
Type: "AWS::Events::Rule"
Properties:
Description: "Rule to trigger lambda"
Name: "TagEvents"
EventPattern: {
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["ec2.amazonaws.com", "rds.amazonaws.com", "lambda.amazonaws.com", "s3.amazonaws.com", "dynamodb.amazonaws.com", "elasticfilesystem.amazonaws.com"],
"eventName": ["CreateVolume", "RunInstances", "CreateImage", "CreateSnapshot", "CreateDBInstance", "CreateFunction20150331", "UpdateFunctionConfiguration20150331v2", "UpdateFunctionCode20150331v2", "CreateBucket", "CreateTable", "CreateMountTarget"]
}
}
State: "ENABLED"
Targets:
- Arn: !GetAtt AutoTaggerLambda.Arn
Id: "TagEventsTargetLambda"
Do I need to add an event to the lambda also? I'm a little confused.
You are missing permissions. An AWS::Lambda::Permission resource gives your EventBridge rule permission to invoke the Lambda. It is added to the Lambda's resource-based policy.
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt AutoTaggerLambda.Arn
Action: 'lambda:InvokeFunction'
Principal: 'events.amazonaws.com'
SourceArn: !GetAtt TagEvents.Arn
Related
I am trying to set up a demo environment to try out SQS as an AWS Event Bridge Source. I tried uploading few documents to SQS to see if Event Bridge detects any change, but I don't see any events triggered. How can I test SQS as a source with AWS Event Bridge?
Resources:
Queue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub ${AWS::StackName}
LambdaHandlerExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
EventConsumerFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.lambda_handler
Role: !GetAtt LambdaHandlerExecutionRole.Arn
Code:
ZipFile: |
import json
def lambda_handler(event, context):
print("Received event: " + json.dumps(event, indent=2))
Runtime: python3.7
Timeout: 50
EventRule:
Type: AWS::Events::Rule
Properties:
Description: eventEventRule
State: ENABLED
EventPattern:
source:
- aws.sqs
resources:
- !GetAtt Queue.Arn
Targets:
- Arn: !GetAtt EventConsumerFunction.Arn
Id: EventConsumerFunctionTarget
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref EventConsumerFunction
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !GetAtt EventRule.Arn
SQS data events (publishing new message) are not source events for Event Bridge (EB). Only management events can be picked up by EB, e.g.:
purging of the queue
creating of new queue
deletion of a queue
Also your event rule should be more generic for that:
EventRule:
Type: AWS::Events::Rule
Properties:
Description: eventEventRule
State: ENABLED
EventPattern:
source:
- aws.sqs
# resources:
# - !GetAtt Queue.Arn
Targets:
- Arn: !GetAtt EventConsumerFunction.Arn
Id: EventConsumerFunctionTarget
You can also enable CloudWatch trial and detect API events for the SQS. This should enable fetching more events.
I might be late but this can benefit someone else,
have a look at this:
https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-ecs-patterns.QueueProcessingFargateService.html
This will handle scaling of the Fargate container based on a number of messages in the SQS Queue.
a simplest stack can be defined using AWS CDK as following:
queue = sqs.Queue(stack, "Queue")
cluster = aws_ecs.Cluster(
stack, 'FargateCluster'
)
queue_processing_fargate_service = QueueProcessingFargateService(stack, "Service",
cluster=cluster,
memory_limit_mi_b=512,
image=ecs.ContainerImage.from_registry("test"),
command=["-c", "4", "amazon.com"],
enable_logging=False,
desired_task_count=2,
environment={
"TEST_ENVIRONMENT_VARIABLE1": "test environment variable 1 value",
"TEST_ENVIRONMENT_VARIABLE2": "test environment variable 2 value"
},
queue=queue,
max_scaling_capacity=5,
container_name="test"
)
I want to export RDS snapshot to s3 using cloudformation which will trigger lambda function. But i'm kind of new to this. Can someone please help me out. I've already written the script for Lambda and SNS but i'm confused that how can I do export RDS snapshot to S3 using cloudformation. I'm attaching my file of Triggering Lambda from SNS. Help will be appreciated. Thanks.
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: >
example-lambda-sns
Example CloudFormation template to subscribe a lambda to an SNS Topic.
Resources:
ExampleTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: !Sub 'example-sns-topic'
TopicName: !Sub 'example-sns-topic'
Subscription:
- Protocol: lambda
Endpoint: !GetAtt ExampleFunction.Arn
ExampleFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: mylambdafunction
Description: mylambdafunction
Handler: index.handler
Runtime: nodejs14.x
Role: arn:aws:iam::48
CodeUri: 's3://testing/tes.zip'
ExampleFunctionInvokePermission:
Type: 'AWS::Lambda::Permission'
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref ExampleFunction
Principal: sns.amazonaws.com
ExampleTopicPolicy:
Type: 'AWS::SNS::TopicPolicy'
Properties:
Topics:
- !Ref ExampleTopic
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: 'sns:Publish'
Resource: !Ref ExampleTopic
Principal:
AWS: '*'
Condition:
ArnLike:
AWS:SourceArn: !Sub 'arn:aws:*:*:${AWS::AccountId}:*'```
following is sample example for lambda function. which will get trigger from sns. SNS will receive notification when snapshot is created.
import json
import boto3
import uuid
import datetime
import os
def lambda_handler(event, context):
snapshotarn = event['Records'][0]['Sns']['Message']
snapshotarn=snapshotarn.split()
snapshotarn=snapshotarn[10].replace('.','')
print(snapshotarn)
rds=boto3.client('rds')
export=rds.start_export_task(
ExportTaskIdentifier='export'+'-'+uuid.uuid4().hex,
SourceArn=snapshotarn,
S3BucketName=os.environ.get('S3_BUCKET_NAME'),
IamRoleArn=os.environ.get('IAM_ROLE_FOR_EXPORT_TASK'),
KmsKeyId=os.environ.get('KMS_KEY_ID'),
S3Prefix=os.environ.get('S3_PREFIX'),
)
status={
'ExportTaskIdentifier':export['ExportTaskIdentifier'],
'S3Bucket':export['S3Bucket'],
'S3Prefix':export['S3Prefix'],
'Status':export['Status'],
'ResponseMetadata':export['ResponseMetadata'],
}
print(status)
# TODO implement
return {
'statusCode': 200,
'body': status
}
I am trying to get my lambda to run when an image is added to a "folder" in an s3 bucket. Here is the template
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: 1. Creates the S# bucket that sotres the images from the camera.\n
2. Resizes the images when a new image shows up from a camera.\n
3. Adds a record of the image in the DB.
Globals:
Function:
Timeout: 10
Parameters:
DeploymentStage:
Type: String
Default: production
Resources:
CameraImagesBucket:
Type: 'AWS::S3::Bucket'
Properties:
BucketName: !Sub
- com.wastack.camera.images.${stage}
- { stage: !Ref DeploymentStage }
CreateThumbnailFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: image_resize/
Handler: app.lambda_handler
Runtime: python3.8
Description: Creates a thumbnail of images in the camare_images bucket
Policies:
- S3ReadPolicy:
BucketName: !Sub
- com.wastack.camera.images.${stage}
- { stage: !Ref DeploymentStage }
- S3WritePolicy:
BucketName: !Sub
- com.wastack.camera.images.${stage}
- { stage: !Ref DeploymentStage }
Events:
CameraImageEvent:
Type: S3
Properties:
Bucket:
Ref: CameraImagesBucket
Events:
- 's3:ObjectCreated:*'
Filter:
S3Key:
Rules:
- Name: prefix
Value: camera_images
When I look at the lambda created on the AWS console, I do not see the trigger even in the lambda visualiser. The lambda doesn't event have the s3 read and write policies attached to it.
The s3 bucket and the lambda are created, but the policies and triggers that are supposed to connect them are not created.
I did not get any error when I run sam deploy
Question: why did it not attach the s3 trigger event or the s3 access policies to the lambda function?
Policies for s3 So the template is straight forward. If you place the full template in does it work. If that is also failing, check the permissions on what you're running SAM as. Also there's an open ticket on github, This appears to be your issue. See comments.
I am trying to create an AWS Eventbridge rule with a Lambda function as a target. I can add the rule and target fine but when I try to set the lambda permissions via RoleArn the Cloudformation stack deployment fails with:
RoleArn is not supported for target arn:aws:lambda:us-east-1:1234567890:function:contacts-lambda-consume-new-customer. (Service: AmazonCloudWatchEvents; Status Code: 400; Error Code: ValidationException; Request ID: xxxxx-ec5d-45e8-b45d-xxxxxx; Proxy: null)
Here is my Cloudformation stack code:
EventRuleNewCustomer:
Type: AWS::Events::Rule
Properties:
Name: new-customer
EventBusName: myEventBus
# RoleArn: !Join ["", ["arn:aws:iam::",!Ref "AWS::AccountId", ":role/my-role"] ] #no error but doesn't add the permissions
Description: "New customer event rule"
EventPattern:
detail-type:
- "NewCustomer"
State: "ENABLED"
Targets:
-
Arn: !Join ["", ["arn:aws:lambda:" ,!Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":function:contacts-lambda-consume-new-customer"] ]
Id: "NewCustomer"
RoleArn: !Join ["", ["arn:aws:iam::",!Ref "AWS::AccountId", ":role/my-role"] ]
I have tried setting a RoleArn on the rule itself which doesn't give an error when the stack is created but also doesn't add the necessary permissions to execute the Lambda.
The work-around I am using is to edit the lambda target in the AWS Eventbridge console. This seems to do some behind the scenes magic to add the correct permissions for Eventbridge to be able to execute the lambda
Any ideas gratefully appreciated.
This seems to do some behind the scenes magic to add the correct permissions for Eventbridge to be able to execute the lambda
In case of lambda, the permissions are set using Lambda's resource-based policy.
Thus you should use AWS::Lambda::Permission in CloudFormation to allow EventBridge to invoke your function, rather than using RoleArn.
So your permissions would be something as the following (just an example):
EventBridgeLambdaPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt function.Arn
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !GetAtt EventRuleNewCustomer.Arn
I am having some trouble writing the cloudformation script for cloudwatch event trigger to kick off my lambda script, I know I can do it through the console but my requirement is that I need to provision everything in cloudformation. I followed the documentation and it still haven't worked for me and I kept getting the error:
Template contains errors.: Invalid template property or properties
[rPermissionForEventsToInvokeLambda, rLambdaScheduledRule]
can someone point out what is the issue with this part of my cloudformation script? I followed the document almost to the letter and still having error, even the example in the documentation have the same error when I tried to validate it. my cloudformation code is below, any help is appreciated!
rLambdaScheduledRule:
Type: AWS::Events::Rule
Properties:
ScheduleExpression: rate(1 hour)
State: ENABLED
Targets:
Ref:
Fn::ImportValue:
Fn::Sub: rUploadLambda
Action: lambda:InvokeFunction
rPermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Ref:
Fn::ImportValue:
Fn::Sub: rUploadLambda
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn:
Fn::GetAtt:
- rLambdaScheduledRule
- Arn
1) You must export the Lambda function ARN in the CloudFormation template in which you create the lambda function. You need to pass the Lambda function ARN as input to the cloudwatch event (The AWS::Events::Rule Targets attribute requires a resource ARN).
See a sample script below:
Resources:
# Create Controlled Lambda Function
myLambda:
Type: "AWS::Lambda::Function"
Properties:
Code:
S3Bucket: "lambda-bucket"
S3Key: "myhandler.zip"
Description: "Lambda handler"
FunctionName: "myhandler"
Handler: myhandler.myhandler
MemorySize: 128
Role: "arn:aws:iam::xxxxxxxxxxx:role/myLambdaExecutionRole-NC7FA7TUSZ5B"
Runtime: "python3.6"
Timeout: 10
# Output of the cf template
Outputs:
myLambdaArn:
Description: Arn of the my_lambda_function
Value: !GetAtt myLambda.Arn
Export:
Name: !Sub "${AWS::StackName}-LambdaArn"
2) When you create the CloudWatch Event, you need to pass the ARN of the lambda function created in Step1 as the Target.
See a sample script below:
Resources:
# Cloudwatch event to trigger lambda periodically
rLambdaScheduledRule:
Type: "AWS::Events::Rule"
Properties:
Description: "CloudWatch Event to trigger lambda fn"
ScheduleExpression: "rate(1 hour)"
State: "ENABLED"
Targets:
-
Arn:
Fn::ImportValue:
!Sub "${NetworkStackName}-LambdaArn"
Id: "targetevent_v1"
PermissionForEventsToInvokeLambda:
Type: "AWS::Lambda::Permission"
Properties:
FunctionName:
Fn::ImportValue:
!Sub "${NetworkStackName}-LambdaArn"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- rLambdaScheduledRule
- Arn
The value of ${NetworkStackName} should be the StackName from Step1.
Some of the issues you need to correct in your template:
correct the Targets property of resource rLambdaScheduledRule.
remove Action property from resource rLambdaScheduledRule.
correct the FunctionName property of resource rPermissionForEventsToInvokeLambda.
Keeping above sample as reference, correct your template and try again.