I'm having a strange behavior with cloudformation template. This my template, where I create a bucket and want to notification configuration depending on a condition :
AWSTemplateFormatVersion: '2010-09-09'
Description: "Setup Artifacts Bucket"
Parameters:
BucketName:
Description: Name of the pipeline setup arctifact bucket
Type: String
Default: "s3-pipeline-setup"
NotificationCondition:
Description: Conditionally add Notification configuration to the artifact bucket
Type: String
Default: false
Conditions:
AddNotificationConfiguration: !Equals [ !Ref NotificationCondition, true ]
Resources:
ArtifactBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref BucketName
Fn::If:
- AddNotificationConfiguration
-
NotificationConfiguration:
LambdaConfigurations:
-
Function: "arn:aws:lambda:eu-west-1:341292222222227:function:lambda-ops-trigger-pipeline-setup"
Event: "s3:ObjectCreated:*"
Filter:
S3Key:
Rules:
-
Name: prefix
Value: "appstackcodes/"
-
Name: suffix
Value: "txt"
- !Ref AWS::NoValue
When I try a deploy it fails with this error :
00:28:10
UTC+0200 CREATE_FAILED AWS::S3::Bucket ArtifactBucket Encountered
unsupported property Fn::If
I don't really understand the matter.. Can someone try and let me know the mistake there please?
Thanks
Unfortunately you can not do what you intended in cloudformation.
The Fn::If can basically just be used as a ternary expression. E.g.
key: Fn::If: [condition_name, value_if_true, value_if_false]
It can't be used as logic flow like you would in a programming language. There are ways around it. You actually already seemed to have discovered the AWS::NoValue, so it's just a matter of moving the NotificationConfiguration assignment to outside the if.
Resources:
ArtifactBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref BucketName
NotificationConfiguration:
Fn::If:
- AddNotificationConfiguration
- LambdaConfigurations:
-
Function: "arn:aws:lambda:eu-west-1:341294322147:function:lambda-itops-trigger-pipeline-setup"
Event: "s3:ObjectCreated:*"
Filter:
S3Key:
Rules:
-
Name: prefix
Value: "appstackcodes/"
-
Name: suffix
Value: "txt"
- !Ref AWS::NoValue
Effectively you are always assigning something to NotificationConfiguration, but sometimes it's the magic AWS::NoValue. This works in the majority of cases, although there are times when this just isn't sufficient and more creativity is required!
Related
I am trying to deploy this CloudFormation template across my organization that contains global resources as well as Region-specific. I have seen several methods pertaining to my issue but none seem to work as I was hoping. I pieced my script together with other StackOverflow answers that I can't find the links to right now, but most pointed to this handy link: https://garbe.io/blog/2017/07/17/cloudformation-hacks/
Here is how I have essentially pieced my script together (broken up for easier reading):
Parameters:
CommercialMaster:
Description: The Commercial Account ID for routing.
Type: String
Default: ############
GovCloudMaster:
Description: The GovCloud Account ID for routing.
Type: String
Default: ############
LambdaRoleName:
Type: String
Default: 'LambdaR53Role'
Conditions:
Commercial: !Equals [ !Ref AWS::Partition, 'aws' ]
RegionCheck: !Or [!Equals [!Ref AWS::Region, 'us-east-1'], !Equals [!Ref AWS::Region, 'us-gov-west-1']]
CreateLambdaRole: !Equals [ !Ref LambdaRoleName, 'false' ]
CreateLambdaRoleRegion: !And
- !Condition RegionCheck
- !Condition CreateLambdaRole
Resources:
LambdaPermissionsRole:
Type: "AWS::IAM::Role"
Condition: RegionCheck
Properties:
RoleName: LambdaR53Role
Description: Lambda permissions role for R53 association.
CreateRoleWaitHandle:
Condition: CreateLambdaRole
DependsOn: LambdaPermissionsRole
Type: AWS::CloudFormation::WaitConditionHandle
#added, since DependsOn: !If is not possible, trigger by WaitCondition if CreateLambdaRole is false
WaitHandle:
Type: AWS::CloudFormation::WaitConditionHandle
#added, since DependsOn: !If is not possible
WaitCondition:
Type: AWS::CloudFormation::WaitCondition
Properties:
Handle: !If [CreateLambdaRole, !Ref CreateRoleWaitHandle, !Ref WaitHandle]
Timeout: "1"
Count: 0
AssociateVPCs:
Type: 'AWS::Lambda::Function'
DependsOn: WaitCondition
Properties:
Obviously, because of the hardcoded AWS::Region this works fine when deploying out to us-east-1 for Commercial, but when deploying with a StackSet, it fails in any other region. If I remove the RegionCheck then the stack fails because it tries to create the Role in every region and realizes that it already exists (because IAM is global, obviously).
My question, the part I'm stuck on, is how I can create the global IAM role, and STILL deploy the lambda function out to each region I need it (4 regions)?
My temporary solution, which I don't like is to essentially hardcode the region on to the back of the role name, like so:
Resources:
LambdaPermissionsRole:
Type: "AWS::IAM::Role"
Condition: CreateLambdaRole
Properties:
RoleName: !Sub LambdaR53Role-${AWS::Region}
Description: Lambda permissions role for R53 association.
Any further guidance would be great, but this headache has given me the push to start examining Terraform as my way forward so things can be more consolidated.
I have attached the sample to give you some clarity of what i am trying to solve.
AWSTemplateFormatVersion: '2010-09-09'
Description: Project Service Catalog get lambda data
Parameters:
Environment:
Type: String
Description: Environment of the SageMaker
ProjectId:
Type: String
Description: Project ID of the SageMaker
SsmRoleLambdaArn:
Type: AWS::SSM::Parameter::Value<String>
Default: '/data-science/role-lambda/arn'
Description: Arn to lookup Role of the Session using project id
Resource:
IdentifyUserRole:
Type: Custom::GetParam
Properties:
ServiceToken: !Ref SsmRoleLambdaArn
pl_role: !Sub '${Environment}-sso-data-science-${ProjectId}-pl-role'
ds_role: !Sub '${Environment}-sso-data-science-${ProjectId}-ds-role'
KmsKey:
Type: AWS::KMS::Key
Properties:
Description: !Sub 'Encryption for ${Environment}-${ProjectId}-${Prid}-${NotebookInstanceNameSuffix}'
EnableKeyRotation: true
Tags:
- Key: Environment
Value: !Ref Environment
- Key: Owner
Value: !Ref Owner
- Key: ProjectId
Value: !Ref ProjectId
- Key: PrincipalId
Value: !Sub
- "${RoleId}:${Prid}"
- RoleId:
Fn::If: [!Equals [!GetAtt IdentifyUserRole.value, true], !GetAtt PORoleId.value, !GetAtt DSRoleId.value]
I am getting error at the IF condition in the PrincipalID tag. Please help solve this condition with some sample templates. I can't use !GetAtt in the Conditions block as well because we are not supposed to use get attributes.
Error Message - During stack validation
An error occurred (ValidationError) when calling the ValidateTemplate operation: Template error: Fn::If requires a list argument with the first element being a condition
You can't hard code the condition in the If like you are attempting:
Fn::If: [!Equals [!GetAtt IdentifyUserRole.value, true], !GetAtt PORoleId.value, !GetAtt DSRoleId.value]
The first argument must be condition from Conditions section (docs):
reference to a condition in the Conditions section.
Subsequently, you can't construct conditions based on GetAtt or any other resources from Resources section.
The same docs also write:
You can only reference other conditions and values from the Parameters and Mappings sections of a template. For example, you can reference a value from an input parameter, but you cannot reference the logical ID of a resource in a condition.
For the sake of not duplicating code, I'd like to be able to create a filter rule with an OR condition... If either condition is true, then that satisfies the filter.
This is the original way that I'd like to avoid.
MyBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: myBuck
NotificationConfiguration:
TopicConfigurations:
- Event: 's3:ObjectCreated:*'
Filter:
S3Key:
Rules:
- Name: prefix
Value: data/abcd/
Topic: myTopicArnGoesHere
- Event: 's3:ObjectCreated:*'
Filter:
S3Key:
Rules:
- Name: prefix
Value: data/efgh/
Topic: myTopicArnGoesHere
instead I'm hoping that there's a way to do something like this
NewBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: myBuck
VersioningConfiguration:
Status: Enabled
NotificationConfiguration:
TopicConfigurations:
- Event: 's3:ObjectCreated:*'
Filter:
S3Key:
Rules:
- Name: prefix
Value: data/abcd/
OR
- Name: prefix
Value: data/efgh/
Topic: myTopicArnGoesHere
The following AWS CloudFormation gives a circular dependency error. My understanding is that the dependencies flow like this: rawUploads -> generatePreview -> previewPipeline -> rawUploads. Although it doesn't seem like rawUploads depends on generatePreview, I guess CF needs to know what lambda to trigger when creating the bucket, even though the trigger is defined in the lambda part of the CloudFormation template.
I've found some resources online that talk about a similar issue, but it doesn't seem to apply here. https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-circular-dependency-cloudformation/
What are my options for breaking this circular dependency chain? Scriptable solutions are viable, but multiple deployments with manual changes are not for my use case.
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Resources:
rawUploads:
Type: 'AWS::S3::Bucket'
previewAudioFiles:
Type: 'AWS::S3::Bucket'
generatePreview:
Type: AWS::Serverless::Function
Properties:
Handler: generatePreview.handler
Runtime: nodejs6.10
CodeUri: .
Environment:
Variables:
PipelineId: !Ref previewPipeline
Events:
BucketrawUploads:
Type: S3
Properties:
Bucket: !Ref rawUploads
Events: 's3:ObjectCreated:*'
previewPipeline:
Type: Custom::ElasticTranscoderPipeline
Version: '1.0'
Properties:
ServiceToken:
Fn::Join:
- ":"
- - arn:aws:lambda
- Ref: AWS::Region
- Ref: AWS::AccountId
- function
- aws-cloudformation-elastic-transcoder-pipeline-1-0-0
Name: transcoderPipeline
InputBucket:
Ref: rawUploads
OutputBucket:
Ref: previewAudioFiles
One way is to give the S3 buckets explicit names so that later, instead of relying on Ref: bucketname, you can simply use the bucket name. That's obviously problematic if you want auto-generated bucket names and in those cases it's prudent to generate the bucket name from some prefix plus the (unique) stack name, for example:
InputBucket: !Join ["-", ['rawuploads', Ref: 'AWS::StackName']]
Another option is to use a single CloudFormation template but in 2 stages - the 1st stage creates the base resources (and whatever refs are not circular) and then you add the remaining refs to the template and do a stack update. Not ideal, obviously, so I would prefer the first approach.
You can also use the first technique in cases when you need a reference to an ARN, for example:
!Join ['/', ['arn:aws:s3:::logsbucket', 'AWSLogs', Ref: 'AWS:AccountId', '*']]
When using this technique, you may want to also consider using DependsOn because you have removed an implicit dependency which can sometimes cause problems.
This post helped me out in the end: https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-destination-s3/
I ended up configuring an SNS topic in CloudFormation. The bucket would push events on this topic, and the Lambda function listens to this topic. This way the dependency graph is as follows:
S3 bucket -> SNS topic -> SNS topic policy
Lambda function -> SNS topic
Lambda function -> transcoder pipeline
Something along the lines of this (some policies omitted)
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Resources:
SNSTopic:
Type: AWS::SNS::Topic
SNSTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Id: MyTopicPolicy
Version: '2012-10-17'
Statement:
- Sid: Statement-id
Effect: Allow
Principal:
AWS: "*"
Action: sns:Publish
Resource:
Ref: SNSTopic
Condition:
ArnLike:
aws:SourceArn:
!Join ["-", ['arn:aws:s3:::rawuploads', Ref: 'AWS::StackName']]
Topics:
- Ref: SNSTopic
rawUploads:
Type: 'AWS::S3::Bucket'
DependsOn: SNSTopicPolicy
Properties:
BucketName: !Join ["-", ['rawuploads', Ref: 'AWS::StackName']]
NotificationConfiguration:
TopicConfigurations:
- Topic:
Ref: "SNSTopic"
Event: 's3:ObjectCreated:*'
previewAudioFiles:
Type: 'AWS::S3::Bucket'
generatePreview:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Join ["-", ['generatepreview', Ref: 'AWS::StackName']]
Handler: generatePreview.handler
Runtime: nodejs6.10
CodeUri: .
Environment:
Variables:
PipelineId: !Ref previewPipeline
Events:
BucketrawUploads:
Type: SNS
Properties:
Topic: !Ref "SNSTopic"
previewPipeline:
Type: Custom::ElasticTranscoderPipeline
DependsOn: 'rawUploads'
Version: '1.0'
Properties:
ServiceToken:
Fn::Join:
- ":"
- - arn:aws:lambda
- Ref: AWS::Region
- Ref: AWS::AccountId
- function
- aws-cloudformation-elastic-transcoder-pipeline-1-0-0
Name: transcoderPipeline
InputBucket:
!Join ["-", ['arn:aws:s3:::rawuploads', Ref: 'AWS::StackName']]
OutputBucket:
Ref: previewAudioFiles
I have a very confusing issue. I am trying to create s3 bucket with an event attached to it trigger lambda. Here is my code:
#s3-test-bucket
---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
project:
Description: project
Type: String
ConstraintDescription: Any string
EnvironmentApp:
Description: EnvironmentApp
Type: String
ConstraintDescription: Any string
S3BucketName:
Description: EnvironmentApp
Type: String
ConstraintDescription: Any string
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub images-${EnvironmentApp}
NotificationConfiguration:
LambdaConfigurations:
-
Function: arn:aws:lambda:us-east-1:xxxxxxxxxxx:function:test-
trigger-cfn
Event: "s3:ObjectCreated:*"
Filter:
S3Key:
Rules:
-
Name: suffix
Value: zip
DeletionPolicy: Delete
Now the problem is when I run it I get the following error:
10:25:56 UTC-0300 CREATE_FAILED AWS::S3::Bucket S3Bucket Unable to validate the following destination configurations
I have my lambda created before running the stack so what could be the cause of the issue?
Update: this fixed the issue :
LambdaPolicy:
DependsOn:
- Lambda
Type: AWS::Lambda::Permission
Properties:
FunctionName:
"Fn::GetAtt": [ LambdaImageResizer, Arn ]
Action: "lambda:InvokeFunction"
Principal: "s3.amazonaws.com"
SourceArn: arn:aws:s3:::xxxxx
Please check if you have allowed invokeFunction permission on your lambda function.
This looks similar issue as stated in AWS forum
https://forums.aws.amazon.com/thread.jspa?threadID=167470