S3 NotificationConfiguration - unable to validate destination configuration - amazon-web-services

I'm trying to setup an S3 bucket which notifies a Lambda function when a new object is created.
The stack below works fine but I want to added the SourceArn to the Lambda permission, following best practices.
There's some literature on this which suggests the way to do it is via a string rather than Fn::GetAtt/Arn -
https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-circular-dependency-cloudformation/
But if I uncomment the relevant SourceArn line and redeploy, I get -
Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: TBZDEVZ1HVD9EN0Q; S3 Extended Request ID: gL3CRz6UayvHup5i5oC4+/RMm0p1oRaRrPVtfZrykeAaJ1BVuhNSKkqxQ8TL5sy749d9PtbMOEQ=; Proxy: null)
Ho hum. Looking at the article again, I see that MyBucket needs to depend on MyBucketFunctionPermission - but if I uncomment that and redeploy I now get -
An error occurred (ValidationError) when calling the CreateChangeSet operation: Circular dependency between resources: [MyBucketFunctionPermission, MyBucket]
This is some fresh circle of hell. Am I missing something from the article or is there some other combination of SourceArn format + DependsOn that would get this to work ?
TIA.
AWSTemplateFormatVersion: '2010-09-09'
Outputs: {}
Parameters: {}
Resources:
MyBucket:
# DependsOn:
# - MyBucketFunctionPermission
Properties:
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:*
Function:
Fn::GetAtt:
- MyBucketFunction
- Arn
Type: AWS::S3::Bucket
MyBucketFunction:
Properties:
Code:
ZipFile: "def handler(event, context):\n print (event)"
Handler: index.handler
Role:
Fn::GetAtt:
- MyBucketRole
- Arn
Runtime: "python3.8"
Type: AWS::Lambda::Function
MyBucketFunctionPermission:
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: MyBucketFunction
Principal: s3.amazonaws.com
# SourceArn:
# Fn::Sub: arn:aws:s3:::${MyBucket}
Type: AWS::Lambda::Permission
MyBucketRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action: logs:*
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName:
Fn::Sub: my-bucket-role-policy-1234567890
Type: AWS::IAM::Role

There is nothing seemingly wrong with your template. It works fine, at least for me. But a race condition is possible between MyBucket and MyBucketFunctionPermission. Thus, protect against this, you have to use DependsOn. But for that to work you have to explicitly set your bucket name. For example:
AWSTemplateFormatVersion: '2010-09-09'
Outputs: {}
Parameters: {}
Resources:
MyBucket:
DependsOn:
- MyBucketFunctionPermission
Properties:
BucketName: !Sub "my-bucket-323323-${AWS::StackName}-${AWS::Region}"
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:*
Function:
Fn::GetAtt:
- MyBucketFunction
- Arn
Type: AWS::S3::Bucket
MyBucketFunction:
Properties:
Code:
ZipFile: "def handler(event, context):\n print (event)"
Handler: index.handler
Role:
Fn::GetAtt:
- MyBucketRole
- Arn
Runtime: "python3.8"
Type: AWS::Lambda::Function
MyBucketFunctionPermission:
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: MyBucketFunction
Principal: s3.amazonaws.com
SourceArn: !Sub "arn:aws:s3:::my-bucket-323323-${AWS::StackName}-${AWS::Region}"
Type: AWS::Lambda::Permission
MyBucketRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action: logs:*
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName:
Fn::Sub: my-bucket-role-policy-1234567890
Type: AWS::IAM::Role

Related

AWS SAM Lambda policy for OpenSearch

I couldn't get a valid IAM policy to work for a Lambda function to OpenSearch.
Replicate:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub ${AWS::StackName}-Replicate
Description: !Sub
- Stack ${StackTagName} Environment ${EnvironmentTagName} Function ${ResourceName}
- ResourceName: DBReplicate
CodeUri: ../src/Replicate
Handler: index.handler
Runtime: nodejs16.x
MemorySize: 3008
Timeout: 30
Tracing: Active
Policies:
- PolicyName: Access
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- es:*
Resource:
- arn:aws:es:eu-west-1:22222222222:domain/mynewdomain
- DomainName: mynewdomain
Events:
MyDynamoDBtable:
Type: DynamoDB
Properties:
Stream: !Ref TableStreamArn
StartingPosition: TRIM_HORIZON
BatchSize: 1
Running sam validate, I'm getting:
Policy at index 0 in the 'Policies' property is not valid
Got the answer from another post, the structure was wrong, the correct way is:
Add inline policy to aws SAM template
QueryFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: query/
Handler: app.lambda_handler
Policies:
- AmazonDynamoDBFullAccess
- AWSLambdaVPCAccessExecutionRole
- Version: '2012-10-17' # Policy Document
Statement:
- Effect: Allow
Action:
- dynamodb:*
Resource: 'arn:aws:dynamodb:*:*:table/dynamo_db_table_endpoint'

Lambda destination not being triggered on Lambda error

I have a Cloudformation stack which connects two Lambdas together via a Lambda destination config and an SQS queue.
The idea is that ErrorsFunction is fired when there is an error in HelloAddFunction.
This stack deploys fine, and HelloAddFunction works fine when I invoke it with some integer values.
I can see an error in HelloAddFunction when I invoke it with non- integer values, but no corresponding error seems to be received by ErrorsFunction.
The binding of ErrorsFunction to ErrorsQueue seems to be working - if I push a message onto ErrorsQueue via the console, it's received by ErrorsFunction.
So it feels like the Lambda destination config is somehow not working.
What am I missing here ?
TIA
AWSTemplateFormatVersion: '2010-09-09'
Outputs: {}
Parameters:
AppName:
Type: String
MemorySizeSmall:
Default: 512
Type: Number
RuntimeVersion:
Default: '3.8'
Type: String
TimeoutShort:
Default: 5
Type: Number
Resources:
HelloAddEventConfig:
Properties:
DestinationConfig:
OnFailure:
Destination:
Fn::GetAtt:
- ErrorsQueue
- Arn
FunctionName:
Ref: HelloAddFunction
MaximumRetryAttempts: 0
Qualifier: $LATEST
Type: AWS::Lambda::EventInvokeConfig
HelloAddFunction:
Properties:
Code:
ZipFile: |
def handler(event, context):
x, y = int(event["x"]), int(event["y"])
return x+y
Handler: index.handler
MemorySize:
Ref: MemorySizeSmall
Role:
Fn::GetAtt:
- HelloAddRole
- Arn
Runtime:
Fn::Sub: python${RuntimeVersion}
Timeout:
Ref: TimeoutShort
Type: AWS::Lambda::Function
HelloAddRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action: logs:*
Effect: Allow
Resource: '*'
- Action: sqs:*
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName:
Fn::Sub: hello-add-role-policy-${AWS::StackName}
Type: AWS::IAM::Role
ErrorsFunction:
Properties:
Code:
ZipFile: |
def handler(event, context):
print (event)
Handler: index.handler
MemorySize:
Ref: MemorySizeSmall
Role:
Fn::GetAtt:
- ErrorsRole
- Arn
Runtime:
Fn::Sub: python${RuntimeVersion}
Timeout:
Ref: TimeoutShort
Type: AWS::Lambda::Function
ErrorsQueue:
Properties: {}
Type: AWS::SQS::Queue
ErrorsQueueBinding:
Properties:
EventSourceArn:
Fn::GetAtt:
- ErrorsQueue
- Arn
FunctionName:
Ref: ErrorsFunction
Type: AWS::Lambda::EventSourceMapping
ErrorsRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action: logs:*
Effect: Allow
Resource: '*'
- Action: sqs:*
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName:
Fn::Sub: errors-role-policy-${AWS::StackName}
Type: AWS::IAM::Role
Your template is fine, but lambda destinations are for asynchronous invocations of your function only. So you have to make such an invocation, which can be done using AWS CLI (--invocation-type Event). AWS Console is synchronous only.
aws lambda invoke --function-name <HelloAddFunctionnaem> --invocation-type Event --payload '{"x": 3, "y": "SSS"}' /dev/stdout

What is SourceArn when create lambda function through cloudformation template

I want to create a lambda function through cloudformation template, I have ConfigurationLambdaRole, ConfigurationLambdaFunction and ConfigurationLambdaInvokePermission, in ConfigurationLambdaInvokePermission section, what should be the SourceArn? And is there any incorrect thing with my template?
Resources:
ConfigurationLambdaRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: 'configuration-lambda'
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- events.amazonaws.com
- s3.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSQSFullAccess
- arn:aws:iam::aws:policy/CloudWatchLogsFullAccess
ConfigurationLambdaFunction:
Type: AWS::Lambda::Function
Properties:
Description: 'configuration service with lambda'
FunctionName: 'configuration-lambda1'
Handler: lambda.handler.EventHandler::handleRequest
Runtime: Java 11
MemorySize: 128
Timeout: 120
Code:
S3Bucket: configurationlambda
S3Key: lambda-service-1.0.0-SNAPSHOT.jar
Role: !GetAtt ConfiguratioLambdaRole.Arn
ConfigurationLambdaInvokePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Fn::GetAtt:
- ConfigurationLambdaFunction
- Arn
Action: 'lambda:InvokeFunction'
Principal: "s3.amazonaws.com"
SourceArn: 'arn of jar file in s3(configurationlambda)'
SourceArn is an arn of a resource which is going to invoke your function. For example, if your lambda would be invoked through S3 Events Notifications, the SourceArn would be the ARN of your bucket.
In your case I don't see why would you need AWS::Lambda::Permission. So I would just remove the entire resource.

Add event on S3 using CloudFormation - 'bucket already exists' error

I am trying to create a lambda function with a S3 trigger. While executing the templates, I am getting S3 bucket already exist error. There is no any buckets with the same name in my S3 and even in this code I am creating bucket only once but somehow it seems it is creating buckets twice.
Below are the my cloudformation templates.
'''python
AWSTemplateFormatVersion : 2010-09-09
Parameters:
BucketName:
Type: String
Resources:
Bucket:
Type: AWS::S3::Bucket
DependsOn:
- ProcessingLambdaPermission
Properties:
BucketName: !Ref BucketName
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:PutObject:*
Function: !GetAtt ProcessingLambdaFunction.Arn
Filter:
S3Key:
Rules:
- Name: suffix
Value: .txt
ProcessingLambdaPermission:
Type: AWS::Lambda::Permission
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref ProcessingLambdaFunction
Principal: s3.amazonaws.com
SourceArn: 'arn:aws:s3:::hope'
SourceAccount: !Ref AWS::AccountId
ProcessingLambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: allowLogging
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:*
Resource: arn:aws:logs:*:*:*
- PolicyName: getAndDeleteObjects
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- s3:GetObject
- s3:DeleteObject
Resource: !Sub 'arn:aws:s3:::${BucketName}/*'
ProcessingLambdaFunction:
Type: AWS::Lambda::Function
Properties:
Code:
ZipFile: !Sub |
import json
import boto3
s3 = boto3.client("s3")
def lambda_handler(event,context):
print("hello")
Handler: index.handler
Role: !GetAtt ProcessingLambdaExecutionRole.Arn
Runtime: python2.7
MemorySize: 512
Timeout: 120
'''

Cloudformation template to push cloudwatch logs to elasticsearch

I am looking for a Cloudformation template to push cloudwatch logs to elasticsearch in another account. Even to the same account would be ok and I can update that.
Seems like a standard problem but haven't seen any template which automates the steps described in https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_ES_Stream.html.
I should have updated the answer. This is what I wrote which is the minimalistic version of streaming logs from CW to Elastic Search.
Resources:
LambdaElasticSearchExecutionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: ['sts:AssumeRole']
Effect: Allow
Principal:
Service: [lambda.amazonaws.com]
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action: ['es:ESHttpPost']
Effect: Allow
Resource: "*"
- Action: ['logs:CreateLogGroup', 'logs:CreateLogStream', 'logs:PutLogEvents']
Effect: Allow
Resource: "arn:aws:logs:*:*:*"
- Action: ['lambda:InvokeFunction']
Effect: Allow
Resource: "arn:aws:logs:*:*:*"
Version: '2012-10-17'
PolicyName: lambdaRoleElasticSearchStreaming
Type: AWS::IAM::Role
ESStreamingLambda:
Type: AWS::Lambda::Function
DependsOn: LambdaElasticSearchExecutionRole
Properties:
Handler: index.handler
Role:
Fn::GetAtt: [LambdaElasticSearchExecutionRole, Arn]
Code:
S3Bucket: {'Fn::Sub': 'do-not-delete-cw-es-log-streaming-lamda-${AWS::Region}'}
S3Key: LogsToElasticsearch.zip
Runtime: nodejs4.3
LambdaPermissionForCWInvokation:
DependsOn: ESStreamingLambda
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName:
Fn::GetAtt: [ESStreamingLambda, Arn]
Principal: {'Fn::Sub': 'logs.${AWS::Region}.amazonaws.com'}
APILogsToElasticSearchSubscriptionFilter:
Type: AWS::Logs::SubscriptionFilter
DependsOn: [ESStreamingLambda, LambdaPermissionForCWInvokation]
Properties:
DestinationArn:
Fn::GetAtt: [ESStreamingLambda, Arn]
FilterPattern: ''
LogGroupName: {Ref: LambdaLogGroup}
```java
I am currently trying to do the same thing. I have found these examples for now:
https://docs.aws.amazon.com/solutions/latest/centralized-logging/templates.html (seems to be the best bet)
https://github.com/amazon-archives/cloudwatch-logs-subscription-consumer (seems to me this is an old approach)
https://github.com/awslabs/hids-cloudwatchlogs-elasticsearch-template (just an example)