CloudFormation Global Resource deployment across multiple regions - amazon-web-services

I am trying to deploy this CloudFormation template across my organization that contains global resources as well as Region-specific. I have seen several methods pertaining to my issue but none seem to work as I was hoping. I pieced my script together with other StackOverflow answers that I can't find the links to right now, but most pointed to this handy link: https://garbe.io/blog/2017/07/17/cloudformation-hacks/
Here is how I have essentially pieced my script together (broken up for easier reading):
Parameters:
CommercialMaster:
Description: The Commercial Account ID for routing.
Type: String
Default: ############
GovCloudMaster:
Description: The GovCloud Account ID for routing.
Type: String
Default: ############
LambdaRoleName:
Type: String
Default: 'LambdaR53Role'
Conditions:
Commercial: !Equals [ !Ref AWS::Partition, 'aws' ]
RegionCheck: !Or [!Equals [!Ref AWS::Region, 'us-east-1'], !Equals [!Ref AWS::Region, 'us-gov-west-1']]
CreateLambdaRole: !Equals [ !Ref LambdaRoleName, 'false' ]
CreateLambdaRoleRegion: !And
- !Condition RegionCheck
- !Condition CreateLambdaRole
Resources:
LambdaPermissionsRole:
Type: "AWS::IAM::Role"
Condition: RegionCheck
Properties:
RoleName: LambdaR53Role
Description: Lambda permissions role for R53 association.
CreateRoleWaitHandle:
Condition: CreateLambdaRole
DependsOn: LambdaPermissionsRole
Type: AWS::CloudFormation::WaitConditionHandle
#added, since DependsOn: !If is not possible, trigger by WaitCondition if CreateLambdaRole is false
WaitHandle:
Type: AWS::CloudFormation::WaitConditionHandle
#added, since DependsOn: !If is not possible
WaitCondition:
Type: AWS::CloudFormation::WaitCondition
Properties:
Handle: !If [CreateLambdaRole, !Ref CreateRoleWaitHandle, !Ref WaitHandle]
Timeout: "1"
Count: 0
AssociateVPCs:
Type: 'AWS::Lambda::Function'
DependsOn: WaitCondition
Properties:
Obviously, because of the hardcoded AWS::Region this works fine when deploying out to us-east-1 for Commercial, but when deploying with a StackSet, it fails in any other region. If I remove the RegionCheck then the stack fails because it tries to create the Role in every region and realizes that it already exists (because IAM is global, obviously).
My question, the part I'm stuck on, is how I can create the global IAM role, and STILL deploy the lambda function out to each region I need it (4 regions)?
My temporary solution, which I don't like is to essentially hardcode the region on to the back of the role name, like so:
Resources:
LambdaPermissionsRole:
Type: "AWS::IAM::Role"
Condition: CreateLambdaRole
Properties:
RoleName: !Sub LambdaR53Role-${AWS::Region}
Description: Lambda permissions role for R53 association.
Any further guidance would be great, but this headache has given me the push to start examining Terraform as my way forward so things can be more consolidated.

Related

AWS Parameter Must Have Values Error (value exists)

I have the following parameter in my CloudFormation script:
CloudFormationURL:
Type: String
Description: S3 URL for nested CloudFormation templates
Default: ""
This parameter covers the CloudFormation scripts in a nested folder of my deployment config.
I use it with a resource like this:
CloudWatchDashboardStack:
Type: "AWS::CloudFormation::Stack"
Properties:
TemplateURL: !Sub "${CloudFormationURL}/cloudwatch-dashboard.cfn.yaml"
Parameters:
AppName: !Ref AppName
DeployPhase: !Ref DeployPhase
DeveloperPrefix: !Ref DeveloperPrefix
Environment: !Ref Environment
Which works fine, and has worked for months.
I needed to add another resource, so I added this:
BatchDNSResources:
Type: "AWS::CloudFormation::Stack"
Properties:
Parameters:
AppName: !Ref AppName
Environment: !Ref Environment
DeveloperPrefix: !Ref DeveloperPrefix
DeployPhase: !Ref DeployPhase
AppVersion: !Ref AppVersion
SharedBucketName: !Ref SharedBucketName
S3Version: !Ref S3Version
HostedZone: !Ref HostedZone
VPCStackName: !FindInMap
- EnvironmentMap
- !Ref Environment
- VpcStackName
Company: !Ref Company
CostCenter: !Ref CostCenter
Team: !Ref Team
TemplateURL: !Sub "${CloudFormationURL}/batch-dns.cfn.yaml"
CloudFormation throws this error and then fails:
Parameters: [CloudFormationURL] must have values
Checking the changeset for the stack I can see the following value for the CloudFormationURL:
s3://application-shared-dev/application-name/qa/cf/nested/KShyDj205UK8mz6W_XUA5TnEF8nqPWHS
Checking the application predeploy logs I can see:
upload: deploy/cloudformation/templates/nested/batch-dns.cfn.yaml to s3://application-shared-dev/application-name/qa/cf/nested/KShyDj205UK8mz6W_XUA5TnEF8nqPWHS/batch-dns.cfn.yaml
And I can see the file in the S3 bucket.
If I remove BatchDNSResource the stack completes successfully.
What the heck am I missing here?
Sometimes, the smallest things will get you.
I had copied the Parameters from the master CloudFormation script, including this one, into the nested script:
CloudFormationURL:
Type: String
Description: S3 URL for nested CloudFormation templates
Default: ""
If you look closely, you will see that I did not pass the parameter into the nested script when calling the resource:
BatchDNSResources:
Type: "AWS::CloudFormation::Stack"
Properties:
Parameters:
AppName: !Ref AppName
Environment: !Ref Environment
DeveloperPrefix: !Ref DeveloperPrefix
DeployPhase: !Ref DeployPhase
AppVersion: !Ref AppVersion
SharedBucketName: !Ref SharedBucketName
S3Version: !Ref S3Version
HostedZone: !Ref HostedZone
VPCStackName: !FindInMap
- EnvironmentMap
- !Ref Environment
- VpcStackName
Company: !Ref Company
CostCenter: !Ref CostCenter
Team: !Ref Team
TemplateURL: !Sub "${CloudFormationURL}/batch-dns.cfn.yaml"
Because the CloudFormation console was saying the issue was with the BatchDNSResources I kept looking at the master script for the problem and missing the reference in the other script. There are two ways to solve this problem:
Keep CloudFormationURL as a parameter in the nested script (if you need it for some reason) and pass the value from the master script.
Remove the parameter from the nested script (if it is not needed)
Sometimes just asking for an extra set of eyeballs and getting a little rest will help you to find the issues. I want to leave this question/answer in place here because when I was searching for the error here and elsewhere no one ever mentioned (probably out of embarrassment) that the mistake is simply overlooking something like this. I hope this answer prompts others to check everything when they run across this type of error.

Creating an AWS log group?

I having issues creating an AWS log group that belongs to a bigger CloudFormation template. So just for testing, I'm creating just the log group with the following template
Parameters:
LogGroupName:
Type: String
Description: 'cloudwatch log group name'
Default: "test-log-group"
LogGroupRetention:
Type: Number
Description: Retention period for log groups in cloudwatch
Default: 30
DelPolicy:
Type: String
Description: 'Deletion policy'
Default: "Retain"
Resources:
LLGO1WY:
Type: 'AWS::Logs::LogGroup'
Properties:
awslogs-region: !Ref 'AWS::Region'
LogGroupName: !Ref LogGroupName
RetentionInDays: !Ref LogGroupRetention
DeletionPolicy: !Ref DelPolicy
When I import the template during the manual stack creation ("Create Stack" button), I get the following when I get to the "Import Overview" page.
There was an error creating this change set
The following resources to import [LLGO1WY] must have DeletionPolicy attribute specified in the template.
If you look the documentation for AWS::Logs::LogGroup, it doesn't even have a DeletionPolicy defined as a property. Note that if I remove that property, I get the same error. Any clues?
AWS::Logs::LogGroup does not have DeletionPolicy property. DeletionPolicy is a top level attribute which you can't parametrize.
I guess you wanted maybe:
Resources:
LLGO1WY:
Type: 'AWS::Logs::LogGroup'
DeletionPolicy: Retain # <--- This is not property and must be here
Properties:
awslogs-region: !Ref 'AWS::Region'
LogGroupName: !Ref LogGroupName
RetentionInDays: !Ref LogGroupRetention

Autoscale ProvisionedConcurrentExecutions AWS Lambda DependsOn value null

Im trying to add auto scaling to the Provisioned Concurrency configuration for our Lambdas on AWS.
Im having trouble with the DependsOn: value for the Lambda target it doesnt seem recognize the resource im passing in and says its null. In the example from AWS its using AutoPublishAlias which isnt an option for me. https://aws.amazon.com/blogs/aws/new-provisioned-concurrency-for-lambda-functions/
Am I referencing incorrectly?
**LambdaAlias:**
Type: AWS::Lambda::Alias
Properties:
Description: Adds provisioned concurrency for lambda using alias on arn
FunctionName: !Ref Lambda
FunctionVersion: !GetAtt LambdaVersion.Version
Name: live
ProvisionedConcurrencyConfig:
ProvisionedConcurrentExecutions: !Ref ProvisionedConcurrentExecutions
LambdaTarget:
Type: AWS::ApplicationAutoScaling::ScalableTarget
Properties:
MaxCapacity: 100
MinCapacity: 1
ResourceId: !Sub function:${Lambda}:live # You need to specify an alis or version here
RoleARN: !Ref IamRoleArn
ScalableDimension: lambda:function:ProvisionedConcurrency
ServiceNamespace: lambda
**DependsOn: LambdaAlias** # This is your function logical ID + "Alias" + what you use for AutoPublishAlias
LambdaTrackingScalingPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyName: utilization
PolicyType: TargetTrackingScaling
ScalingTargetId: !Ref LambdaTarget
TargetTrackingScalingPolicyConfiguration:
TargetValue: 0.70 # Any value between 0.1 and 0.9 can be used here
PredefinedMetricSpecification:
PredefinedMetricType: LambdaProvisionedConcurrencyUtilization
Adding Resource Snippet
Resources:
Lambda:
Type: AWS::Lambda::Function
Properties:
!If
- EnableVpcConfig
- FunctionName: !Ref FunctionName
Description: !Ref FunctionDescription
Code:
S3Bucket: !Ref CodeSourceBucket
S3Key: !Sub 'api-packages/${CodeFile}'
Handler: !Ref Handler
Environment:
Variables:
APP_ID: !If [ UsesPublicKeyAPI, !Ref AppId, !Ref
Solution
Answer was to reference the logical id of the AWS::Lambda::Alias resource I created which in this case was
DependsOn: LambdaAlias
The doc was using AutoPublishAlias which didnt require the creation of a resource AWS::Lambda::Alias so config needed is slightly different
Have you tried just using the name of the resource for the LambdaAlias
I feel DependsOn: LambdaAlias should do the trick
You are using plain Cloudformation where the article you refer to is using AWS SAM.
since you are not using any AutoPublishAlias the format
{functionLogicalResourceName}Alias{aliasName}
should work for you..
In the sample you have provided. You have not provided the part which is deploying Lambda so if we assume your lambda logical id is MyLambda you should
DependsOn: MyLambdaAliaslive

Work around circular dependency in AWS CloudFormation

The following AWS CloudFormation gives a circular dependency error. My understanding is that the dependencies flow like this: rawUploads -> generatePreview -> previewPipeline -> rawUploads. Although it doesn't seem like rawUploads depends on generatePreview, I guess CF needs to know what lambda to trigger when creating the bucket, even though the trigger is defined in the lambda part of the CloudFormation template.
I've found some resources online that talk about a similar issue, but it doesn't seem to apply here. https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-circular-dependency-cloudformation/
What are my options for breaking this circular dependency chain? Scriptable solutions are viable, but multiple deployments with manual changes are not for my use case.
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Resources:
rawUploads:
Type: 'AWS::S3::Bucket'
previewAudioFiles:
Type: 'AWS::S3::Bucket'
generatePreview:
Type: AWS::Serverless::Function
Properties:
Handler: generatePreview.handler
Runtime: nodejs6.10
CodeUri: .
Environment:
Variables:
PipelineId: !Ref previewPipeline
Events:
BucketrawUploads:
Type: S3
Properties:
Bucket: !Ref rawUploads
Events: 's3:ObjectCreated:*'
previewPipeline:
Type: Custom::ElasticTranscoderPipeline
Version: '1.0'
Properties:
ServiceToken:
Fn::Join:
- ":"
- - arn:aws:lambda
- Ref: AWS::Region
- Ref: AWS::AccountId
- function
- aws-cloudformation-elastic-transcoder-pipeline-1-0-0
Name: transcoderPipeline
InputBucket:
Ref: rawUploads
OutputBucket:
Ref: previewAudioFiles
One way is to give the S3 buckets explicit names so that later, instead of relying on Ref: bucketname, you can simply use the bucket name. That's obviously problematic if you want auto-generated bucket names and in those cases it's prudent to generate the bucket name from some prefix plus the (unique) stack name, for example:
InputBucket: !Join ["-", ['rawuploads', Ref: 'AWS::StackName']]
Another option is to use a single CloudFormation template but in 2 stages - the 1st stage creates the base resources (and whatever refs are not circular) and then you add the remaining refs to the template and do a stack update. Not ideal, obviously, so I would prefer the first approach.
You can also use the first technique in cases when you need a reference to an ARN, for example:
!Join ['/', ['arn:aws:s3:::logsbucket', 'AWSLogs', Ref: 'AWS:AccountId', '*']]
When using this technique, you may want to also consider using DependsOn because you have removed an implicit dependency which can sometimes cause problems.
This post helped me out in the end: https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-destination-s3/
I ended up configuring an SNS topic in CloudFormation. The bucket would push events on this topic, and the Lambda function listens to this topic. This way the dependency graph is as follows:
S3 bucket -> SNS topic -> SNS topic policy
Lambda function -> SNS topic
Lambda function -> transcoder pipeline
Something along the lines of this (some policies omitted)
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Resources:
SNSTopic:
Type: AWS::SNS::Topic
SNSTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Id: MyTopicPolicy
Version: '2012-10-17'
Statement:
- Sid: Statement-id
Effect: Allow
Principal:
AWS: "*"
Action: sns:Publish
Resource:
Ref: SNSTopic
Condition:
ArnLike:
aws:SourceArn:
!Join ["-", ['arn:aws:s3:::rawuploads', Ref: 'AWS::StackName']]
Topics:
- Ref: SNSTopic
rawUploads:
Type: 'AWS::S3::Bucket'
DependsOn: SNSTopicPolicy
Properties:
BucketName: !Join ["-", ['rawuploads', Ref: 'AWS::StackName']]
NotificationConfiguration:
TopicConfigurations:
- Topic:
Ref: "SNSTopic"
Event: 's3:ObjectCreated:*'
previewAudioFiles:
Type: 'AWS::S3::Bucket'
generatePreview:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Join ["-", ['generatepreview', Ref: 'AWS::StackName']]
Handler: generatePreview.handler
Runtime: nodejs6.10
CodeUri: .
Environment:
Variables:
PipelineId: !Ref previewPipeline
Events:
BucketrawUploads:
Type: SNS
Properties:
Topic: !Ref "SNSTopic"
previewPipeline:
Type: Custom::ElasticTranscoderPipeline
DependsOn: 'rawUploads'
Version: '1.0'
Properties:
ServiceToken:
Fn::Join:
- ":"
- - arn:aws:lambda
- Ref: AWS::Region
- Ref: AWS::AccountId
- function
- aws-cloudformation-elastic-transcoder-pipeline-1-0-0
Name: transcoderPipeline
InputBucket:
!Join ["-", ['arn:aws:s3:::rawuploads', Ref: 'AWS::StackName']]
OutputBucket:
Ref: previewAudioFiles

AWS Cloudformation interprets conditionnal function as a resource property

I'm having a strange behavior with cloudformation template. This my template, where I create a bucket and want to notification configuration depending on a condition :
AWSTemplateFormatVersion: '2010-09-09'
Description: "Setup Artifacts Bucket"
Parameters:
BucketName:
Description: Name of the pipeline setup arctifact bucket
Type: String
Default: "s3-pipeline-setup"
NotificationCondition:
Description: Conditionally add Notification configuration to the artifact bucket
Type: String
Default: false
Conditions:
AddNotificationConfiguration: !Equals [ !Ref NotificationCondition, true ]
Resources:
ArtifactBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref BucketName
Fn::If:
- AddNotificationConfiguration
-
NotificationConfiguration:
LambdaConfigurations:
-
Function: "arn:aws:lambda:eu-west-1:341292222222227:function:lambda-ops-trigger-pipeline-setup"
Event: "s3:ObjectCreated:*"
Filter:
S3Key:
Rules:
-
Name: prefix
Value: "appstackcodes/"
-
Name: suffix
Value: "txt"
- !Ref AWS::NoValue
When I try a deploy it fails with this error :
00:28:10
UTC+0200 CREATE_FAILED AWS::S3::Bucket ArtifactBucket Encountered
unsupported property Fn::If
I don't really understand the matter.. Can someone try and let me know the mistake there please?
Thanks
Unfortunately you can not do what you intended in cloudformation.
The Fn::If can basically just be used as a ternary expression. E.g.
key: Fn::If: [condition_name, value_if_true, value_if_false]
It can't be used as logic flow like you would in a programming language. There are ways around it. You actually already seemed to have discovered the AWS::NoValue, so it's just a matter of moving the NotificationConfiguration assignment to outside the if.
Resources:
ArtifactBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref BucketName
NotificationConfiguration:
Fn::If:
- AddNotificationConfiguration
- LambdaConfigurations:
-
Function: "arn:aws:lambda:eu-west-1:341294322147:function:lambda-itops-trigger-pipeline-setup"
Event: "s3:ObjectCreated:*"
Filter:
S3Key:
Rules:
-
Name: prefix
Value: "appstackcodes/"
-
Name: suffix
Value: "txt"
- !Ref AWS::NoValue
Effectively you are always assigning something to NotificationConfiguration, but sometimes it's the magic AWS::NoValue. This works in the majority of cases, although there are times when this just isn't sufficient and more creativity is required!