I want to create a lambda function through cloudformation template, I have ConfigurationLambdaRole, ConfigurationLambdaFunction and ConfigurationLambdaInvokePermission, in ConfigurationLambdaInvokePermission section, what should be the SourceArn? And is there any incorrect thing with my template?
Resources:
ConfigurationLambdaRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: 'configuration-lambda'
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- events.amazonaws.com
- s3.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSQSFullAccess
- arn:aws:iam::aws:policy/CloudWatchLogsFullAccess
ConfigurationLambdaFunction:
Type: AWS::Lambda::Function
Properties:
Description: 'configuration service with lambda'
FunctionName: 'configuration-lambda1'
Handler: lambda.handler.EventHandler::handleRequest
Runtime: Java 11
MemorySize: 128
Timeout: 120
Code:
S3Bucket: configurationlambda
S3Key: lambda-service-1.0.0-SNAPSHOT.jar
Role: !GetAtt ConfiguratioLambdaRole.Arn
ConfigurationLambdaInvokePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Fn::GetAtt:
- ConfigurationLambdaFunction
- Arn
Action: 'lambda:InvokeFunction'
Principal: "s3.amazonaws.com"
SourceArn: 'arn of jar file in s3(configurationlambda)'
SourceArn is an arn of a resource which is going to invoke your function. For example, if your lambda would be invoked through S3 Events Notifications, the SourceArn would be the ARN of your bucket.
In your case I don't see why would you need AWS::Lambda::Permission. So I would just remove the entire resource.
Cloudtrail default logs can be streamed to elasticsearch domain as shown in this image. How do I achieve this using cloudformation template?
Update:
If you are using aws-cli, take a look at my answer here.
Well, after a few hours of exploring and reading a lot of documentation I finally succeeded to create this template.
Designer Overview :
In order to enable the stream logs to elasticsearch we need to create the following resources:
The lambda function will forward the logs from cloudwatch log group to Elasticsearch.
Relevant IAM Role to get logs from cloudwatch and insert to Elasticsearch.
Lambda permission - The AWS::Lambda::Permission resource grants an AWS service or another account permission to use a function to allow the cloudwatch log group to trigger the lambda.
Subscription Filter - The AWS::Logs::SubscriptionFilter resource specifies a subscription filter and associates it with the specified log group. Subscription filters allow you to subscribe to a real-time stream of log events and have them delivered to a specific destination.
Template usage:
Download LogsToElasticsearch.zip from my Github page.
Update var endpoint = '${Elasticsearch_Endpoint}'; in index.js with your Elasticseatch url e.g - 'search-xxx-yyyy.eu-west-1.es.amazonaws.com';.
Copy the zip file to s3 bucket which will be used in the template (LambdaArtifactBucketName).
Fill relevant Parameters - you can find descriptions to each resource.
Template YAML:
AWSTemplateFormatVersion: 2010-09-09
Description: Enable logs to elasticsearch
Parameters:
ElasticsearchDomainName:
Description: Name of the Elasticsearch domain that you want to insert logs to
Type: String
Default: amitb-elastic-domain
CloudwatchLogGroup:
Description: Name of the log group you want to subscribe
Type: String
Default: /aws/eks/amitb-project/cluster
LambdaName:
Description: Name of the lambda function
Type: String
Default: amitb-cloudwatch-logs
LambdaRole:
Description: Name of the role used by the lambda function
Type: String
Default: amit-cloudwatch-logs-role
LambdaArtifactBucketName:
Description: The bucket where the lambda function located
Type: String
Default: amit-bucket
LambdaArtifactName:
Description: The name of the lambda zipped file
Type: String
Default: LogsToElasticsearch.zip
VPC:
Description: Choose which VPC the Lambda-functions should be deployed to
Type: 'AWS::EC2::VPC::Id'
Default: vpc-1111111
Subnets:
Description: Choose which subnets the Lambda-functions should be deployed to
Type: 'List<AWS::EC2::Subnet::Id>'
Default: 'subnet-123456789,subnet-123456456,subnet-123456741'
SecurityGroup:
Description: Select the Security Group to use for the Lambda-functions
Type: 'List<AWS::EC2::SecurityGroup::Id>'
Default: 'sg-2222222,sg-12345678'
Resources:
ExampleInvokePermission:
Type: 'AWS::Lambda::Permission'
DependsOn: ExampleLambdaFunction
Properties:
FunctionName:
'Fn::GetAtt':
- ExampleLambdaFunction
- Arn
Action: 'lambda:InvokeFunction'
Principal: !Sub 'logs.${AWS::Region}.amazonaws.com'
SourceArn: !Sub >-
arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CloudwatchLogGroup}:*
SourceAccount: !Ref 'AWS::AccountId'
LambdaExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Ref LambdaRole
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: lambda-to-es-via-vpc-policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'es:*'
Resource:
- !Sub >-
arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ElasticsearchDomainName}
- PolicyName: logs-and-ec2-permissions
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'ec2:CreateNetworkInterface'
- 'ec2:DescribeNetworkInterfaces'
- 'ec2:DeleteNetworkInterface'
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: '*'
ExampleLambdaFunction:
Type: 'AWS::Lambda::Function'
DependsOn: LambdaExecutionRole
Properties:
Code:
S3Bucket: !Ref LambdaArtifactBucketName
S3Key: !Ref LambdaArtifactName
FunctionName: !Ref LambdaName
Handler: !Sub '${LambdaName}.handler'
Role:
'Fn::GetAtt':
- LambdaExecutionRole
- Arn
Runtime: nodejs8.10
Timeout: '300'
VpcConfig:
SecurityGroupIds: !Ref SecurityGroup
SubnetIds: !Ref Subnets
MemorySize: 512
SubscriptionFilter:
Type: 'AWS::Logs::SubscriptionFilter'
DependsOn: ExampleInvokePermission
Properties:
LogGroupName: !Ref CloudwatchLogGroup
FilterPattern: '[host, ident, authuser, date, request, status, bytes]'
DestinationArn:
'Fn::GetAtt':
- ExampleLambdaFunction
- Arn
Results:
Cloudwatch log:
Hope you find it helpfull.
Update 02/09/2020:
node.js 8.10 is now deprecated, you should use node.js 10 or 12.
https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-policy.html
Here is example for lambda invocation by event:
EventRule:
Type: AWS::Events::Rule
Properties:
Description: "EventRule"
EventPattern:
source:
- "aws.ec2"
detail-type:
- "EC2 Instance State-change Notification"
detail:
state:
- "stopping"
State: "ENABLED"
Targets:
-
Arn:
Fn::GetAtt:
- "LambdaFunction"
- "Arn"
Id: "TargetFunctionV1"
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Ref: "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- "EventRule"
- "Arn"
How to modify this tp be able to invoke state machine not lambda? I have replaced "LambdaFunction with "MyStateMachine"
but how modify PermissionForEventsToInvokeLambda? There is no AWS::StepFunctions::Permission in cloudformation :-(
Here's an snippet from the template we used to have to trigger state machine processing from an S3 event:
InboundBucketPutObjectRule:
Type: 'AWS::Events::Rule'
Properties:
Name: !Sub 'inbound-bucket-put-object-rule'
EventPattern:
source:
- aws.s3
detail-type:
- AWS API Call via CloudTrail
detail:
eventSource:
- s3.amazonaws.com
eventName:
- PutObject
- CopyObject
- CompleteMultipartUpload
requestParameters:
bucketName:
- !Ref InboundBucket
Targets:
- Id: ProcessNewObject
Arn: !Ref StateMachine
RoleArn: !GetAtt
- StateMachineStartExecutionRole
- Arn
StateMachineStartExecutionPolicy:
Type: 'AWS::IAM::ManagedPolicy'
Properties:
ManagedPolicyName: !Sub 'state-machine-start-execution'
Roles:
- !Ref StateMachineStartExecutionRole
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'states:StartExecution'
Resource:
- !Ref StateMachine
StateMachineStartExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Sub 'state-machine-start-execution'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: 'sts:AssumeRole'
StateMachine:
Type: 'AWS::StepFunctions::StateMachine'
Properties:
StateMachineName: !Sub 'state-machine'
Taken from real template so I had to anonymize this which might have introduced errors.
Template structure:
InboundBucketPutObjectRule is the CloudWatch event which gets triggered on file uploads.
StateMachineStartExecutionPolicy + StateMachineStartExecutionRole essentially allow CloudWatch event to start execution of a state machine.
StateMachine - the state machine which should be started (definition omitted).
I wrote above that we used to have this - but not anymore. Please see the following answer for our current solution:
https://stackoverflow.com/a/57563395/303810
I'm trying to implement this simple architecture with Serverless Framework :
File Upload to S3 Bucket --> SNS Topic --> 2 Lambda Functions
Actually i'm just testing with just 1 lambda function and here is my code:
service: MyImageLibrary
provider:
name: aws
runtime: python3.6
stage: dev
region: us-west-2
package:
individually: true
functions:
handler: index.lambda_handler
resources:
Resources:
ImageUploadedTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: SNSTopicImageUploadedTopic
ImageUploadedTopicSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint:
Fn::GetAtt: [ TestLambdaFunction , "Arn" ]
Protocol: lambda
TopicArn:
arn:aws:sns:us-west-2:xxxxxxxxxxx:SNSTopicImageUploadedTopic
ImageUploadedTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: AllowBucketToPushNotificationEffect
Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sns:Publish
Resource: "*"
Topics:
- arn:aws:sns:us-west-2:xxxxxxxxxxx:SNSTopicImageUploadedTopic
MyImagesBucket:
Type: AWS::S3::Bucket
DependsOn: ImageUploadedTopicPolicy
Properties:
BucketName: ${self:custom.bucketName}
NotificationConfiguration:
TopicConfigurations:
- Event: s3:ObjectCreated:*
Topic:
arn:aws:sns:us-west-2:xxxxxxxxxxxx:SNSTopicImageUploadedTopic
CorsConfiguration:
CorsRules:
- AllowedMethods:
- GET
- PUT
- POST
- HEAD
AllowedOrigins:
- "*"
AllowedHeaders:
- "*"
In the console i can see that the Lambda function is subscribed to the the Topic :
But in the Lambda Function the SNS topic is not set in the Trigger section :
Am i missing something ?
Ok, i GOT IT !
I had to add another permission to allow the Lambda function to be triggered by the SNS Topic :
TestLambdaFunctionPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
Principal: sns.amazonaws.com
SourceArn:
Ref: ImageUploadedTopic
FunctionName:
Fn::GetAtt: [ TestLambdaFunction, "Arn" ]
And now in the console i can the SNS as trigger for my Lambda function :)
I'm working on setting up my Java AWS lambda functions to be deployed via Codepipeline -> Cloudformation and am having some difficulty with Cloudformation. I've worked with Terraform before, so I understand the general concepts...
To clarify, my code is housed in a Codecommit repository and everything was setup by Codestar, so it created a Codepipeline with a single stage, two-step deployment (generate changeset, execute changeset).
For right now, I am just marking up the sample template.yml file that Codestar created in the repository, hence the HelloWorld references.
In addition to the template.yml file, I also have a buildspec.yml file for Codebuild, though the build process completes successfully.
Below is my template.yml cloudformation script. The ChangeSet step in the Codepipeline deployment stage completes successfully, however the ExecuteChangeset step fails, with "No reason provided" (super helpful). Clicking on the details link brings me to the Cloudformation page for the execute step which does not actually show any errors. It shows a few of the add/remove steps I would expect to see, though not all of the ones I would think would need to happen. If I click "Execute", it fails with the following error:
Error: Failed to execute change set: ChangeSet [arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/awscodestar-test2-lambda/07e71ee0-6a73-11e7-bee5-50d5cd24fac6] cannot be executed in its current execution status of [EXECUTE_FAILED]
What am I doing wrong here? I don't have a good grasp of the Fn::GetAtt call, but I've tried that a few different ways with no joy.
**In addition to identifying what's going wrong, I have two questions:
Please explain what exactly I'm supposed to reference in the Fn::GetAtt function call? Is it the resource name I provide at the top of the resource I'm trying to call (e.g. GetHelloWorld)? Or an explicit name that's provided as a property of that resource (i.e. FunctionName)?
In the Lambda function declaration, I'm trying to setup the Event trigger in-line, which then needs to reference the Lambda function. Can I refer to the Lambda function resource from within the Event declaration that's nested within the Lambda function resource??
Below is my template.yml file.
AWSTemplateFormatVersion: 2010-09-09
Transform:
- AWS::Serverless-2016-10-31
- AWS::CodeStar
Parameters:
ProjectId:
Type: String
Description: AWS CodeStar projectID used to associate new resources to team members
Resources:
RoleForLambda:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service: "lambda.amazonaws.com"
Action: "sts:AssumeRole"
Policies:
- PolicyName: s3put
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
- 's3:PutObject'
Resource:
- 'arn:aws:logs:*:*:*'
- 'arn:aws:s3:*'
GetHelloWorld:
Type: AWS::Serverless::Function
Properties:
Handler: com.aws.codestar.projecttemplates.handler.HelloWorldHandler
Runtime: java8
Timeout: 60
MemorySize: 256
Role:
'Fn::GetAtt':
- RoleForLambda
- Arn
ScheduleRule:
Type: 'AWS::Events::Rule'
Properties:
Name: DownloadFiles
ScheduleExpression: 'cron(2,7,12,17,22,27,32,37,42,47,52,57 * * * ? *)'
State: ENABLED
Targets:
- Arn:
'Fn::GetAtt':
- GetHelloWorld
- Arn
Id: downloadFiles
LambdaInvokePermission:
Type: "AWS::Lambda::Permission"
Properties:
Action: lambda:InvokeFunction
FunctionName: GetHelloWorld
Principal: events.amazonaws.com
SourceAccount: AWS::XXXXXXXXXXXX
SourceArn:
- Arn:
'Fn::GetAtt':
- ScheduleRule
- Arn
In case anyone else gets to this with similar issues. Turns out, I had a few syntax errors and, I'm sure, other problems... Here is a working template.
AWSTemplateFormatVersion: 2010-09-09
Description: >-
This Lambda function does something
Parameters:
ProjectId:
Description: AWS CodeStar projectID used to associate new resources to team members
Type: String
Resources:
DownloadRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: 'sts:AssumeRole'
Policies:
- PolicyName: PutS3Policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
- 's3:PutObject'
- 's3:PutObjectAcl'
- 's3:PutObjectTagging'
- 'sns:Publish'
Resource:
- 'arn:aws:logs:*:*:*'
- 'arn:aws:s3:::myBucket'
- 'arn:aws:s3:::myBucket/*'
- 'arn:aws:sns:us-east-1:xxxxxxxxxxxx:SNS_TOPIC'
Path: /
DownloadFunction:
Type: 'AWS::Lambda::Function'
Properties:
Handler: 'com.mycompany.download.LambdaFunction::lambdaHandler'
MemorySize: '256'
Description: A scheduled Lambda function
FunctionName: Download
Role: !GetAtt
- DownloadRole
- Arn
Runtime: java8
Timeout: '60'
DependsOn:
- DownloadRole
ScheduleRule:
Type: 'AWS::Events::Rule'
Properties:
Name: DownloadFiles
ScheduleExpression: 'cron(2,7,12,17,22,27,32,37,42,47,52,57 * * * ? *)'
State: ENABLED
Targets:
- Arn: !GetAtt
- DownloadFunction
- Arn
Id: DownloadFiles
DependsOn:
- DownloadFunction
LambdaInvokePermission:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName: !GetAtt
- DownloadFunction
- Arn
Action: 'lambda:InvokeFunction'
Principal: events.amazonaws.com
SourceArn: !GetAtt
- ScheduleRule
- Arn
DependsOn:
- DownloadFunction
- ScheduleRule
Maybe not exactly the same issue
This was our error message:
An error occurred (InvalidChangeSetStatus) when calling the ExecuteChangeSet operation: ChangeSet [arn:aws:cloudformation:....]
cannot be executed in its current execution status of [OBSOLETE]
And to resolve I simply reran the same Cloudformation script. Strange enough - this issue occurred in all environments (DEV, TETS, and PROD) and the same step for all 3 situations.