Up until now, for my project, I had just a single cloudformation stack template. However, I recently encountered that limit for the number of bytes you can have in a stack template, which is why I've been looking into nested stacks. I'm having a bit of trouble wrapping my head around the formatting though, because I've seen a lot of differences in the examples provided.
Below is a snippet of the original (non-nested) stack template before hitting the limit.
---
AWSTemplateFormatVersion: "2010-09-09"
Description: "Template for wgs-pipeline"
Parameters:
CloudspanLambdaFuncS3BucketName:
Type: String
CloudspanLambdaFuncS3KeyName:
Default: 'sfn.deployable.zip'
Type: String
CloudspanLambdaFuncModuleName:
Default: 'cloudspan'
Type: String
Resources:
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
InternetGateway:
Type: AWS::EC2::InternetGateway
RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref 'VPC'
VPCGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref 'VPC'
InternetGatewayId: !Ref 'InternetGateway'
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: EC2 Security Group for instances launched in
the VPC by Batch
VpcId: !Ref 'VPC'
# Lambda Resources (the section I want to place in a separate stack)
CloudspanLambdaExecutionRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: "sts:AssumeRole"
Policies:
- PolicyName: CanListBuckets
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "s3:GetBucketLocation"
- "s3:ListAllMyBuckets"
Resource: "arn:aws:s3:::*"
- PolicyName: CanLog
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:*
Resource: arn:aws:logs:*:*:*
CloudspanLambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Handler:
Fn::Join: [ ".", [ Ref: CloudspanLambdaFuncModuleName, "handler"] ]
Role:
Fn::GetAtt: [ CloudspanLambdaExecutionRole, Arn ]
Code:
S3Bucket:
Ref: CloudspanLambdaFuncS3BucketName
S3Key:
Ref: CloudspanLambdaFuncS3KeyName
Runtime: "python3.6"
Timeout: "60"
What I want to do is isolate everything under the "Lambda resources" comment into a separate stack ("lambda stack") and then have this master stack call to that separate "lambda stack".
Below is my current set-up attempt, but I don't know if I'm doing it correctly:
Master template:
---
AWSTemplateFormatVersion: "2010-09-09"
Description: "Master template for wgs-pipeline. Contains network resources, lambda parameters, and batch
parameters"
Parameters:
CloudspanLambdaFuncS3BucketName:
Type: String
CloudspanLambdaFuncS3KeyName:
Default: 'sfn.deployable.zip'
Type: String
CloudspanLambdaFuncModuleName:
Default: 'cloudspan'
Type: String
Resources:
# The stuff that was already in the Resources section in the original stack
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
InternetGateway:
Type: AWS::EC2::InternetGateway
RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref 'VPC'
VPCGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref 'VPC'
InternetGatewayId: !Ref 'InternetGateway'
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: EC2 Security Group for instances launched in the VPC by Batch
VpcId: !Ref 'VPC'
# The section that is referencing a separate stack
LambdaStack:
Type: AWS::CloudFormation::Stack
Properties:
Parameters:
LambdaFunction:
# ref?
LambdaExecutionRole:
# ref?
TemplateURL: # URL to S3 location of lambda stack
TimeoutInMinutes: '60'
LambdaStack template:
---
AWSTEmplateFormatVersion: '2010-09-09'
Description: Lambda functions stack, contains lambda function and lambda function execution roles.
Parameters:
LambdaFunction:
Description: the lambda function.
LambdaExecutionRole:
Description: the lambda execution roles.
Resources:
Type: "AWS::Lambda::Function"
Properties:
Handler:
#
# Need to figure out how to reference these since they are parameters that exist in the master template
#
Fn::Join: [ ".", [ Ref: CloudspanLambdaFuncModuleName, "handler"] ]
Role:
Fn::GetAtt: [ CloudspanLambdaExecutionRole, Arn ]
Code:
S3Bucket:
Ref: CloudspanLambdaFuncS3BucketName
S3Key:
Ref: CloudspanLambdaFuncS3KeyName
Runtime: "python3.6"
Timeout: "60" # This will be included in the master stack?
Outputs:
LambdaStack:
Description: Lambda stack ID.
Value:
Ref: LambdaFunction
Export:
Name:
Fn::Sub: "${AWS::StackName}-LambdaFunction"
LambdaExecutionRoleStack:
Description: Lambda execution role stack ID.
Value:
Ref: LambdaExecutionRole
Export:
Name:
Fn::Sub: "${AWS::StackName}-LambdaExecutionRole"
Have I formatted the LambdaStack template and master template correctly so far? An in terms of the CloudspanLambdaFunction and CloudspanLambdaExecutionRole parameters that were in the original (un-nested) stack, how exactly do I format those in the LambdaStack (nested) template? Would it be just adding the two parameters under the resources section of the LambdaStack yaml?
Related
I am trying to set-up a websocket gateway to a Lambda function in AWS. When I do this manually I can successfully deploy the websocket and try it out using wscat. However I would like to build the architecture up using CloudFormation.
The structure of my CloudFormation yaml file looks like this:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
lambdaRole:
Type: String
Default: ...
backendRole:
Type: String
Default: ...
lambdaImage:
Type: String
Default: ...
Resources:
MyLambdaFunction:
Type: AWS::Lambda::Function
Properties:
Code:
ImageUri: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${backendRole}:${lambdaImage}
Description: lambda connect function
FunctionName: myLambdaFunction
MemorySize: 128
Role: !Sub arn:aws:iam::${AWS::AccountId}:role/${lambdaRole}
Timeout: 3
PackageType: Image
MyWebSocket:
Type: AWS::ApiGatewayV2::Api
Properties:
Name: MyWebSocket
ProtocolType: WEBSOCKET
RouteSelectionExpression: $request.body.action
MyIntegration:
Type: AWS::ApiGatewayV2::Integration
Properties:
ApiId: !Ref MyWebSocket
Description: Lambda Integration
IntegrationType: AWS_PROXY
IntegrationUri: !Join
- ''
- - 'arn:'
- !Ref 'AWS::Partition'
- ':apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !GetAtt MyLambdaFunction.Arn
- /invocations
IntegrationMethod: POST
MyConnectRoute:
Type: AWS::ApiGatewayV2::Route
Properties:
ApiId: !Ref MyWebSocket
RouteKey: $connect
Target: !Join
- /
- - integrations
- !Ref MyIntegration
MyDefaultRoute:
Type: AWS::ApiGatewayV2::Route
Properties:
ApiId: !Ref MyWebSocket
RouteKey: $default
Target: !Join
- /
- - integrations
- !Ref MyIntegration
MyResponseRoute:
Type: AWS::ApiGatewayV2::Route
Properties:
ApiId: !Ref MyWebSocket
RouteKey: add
RouteResponseSelectionExpression: $default
Target: !Join
- /
- - integrations
- !Ref MyIntegration
MyRouteResponse:
Type: AWS::ApiGatewayV2::RouteResponse
Properties:
RouteId: !Ref MyResponseRoute
ApiId: !Ref MyWebSocket
RouteResponseKey: $default
MyIntegrationResponse:
Type: AWS::ApiGatewayV2::IntegrationResponse
Properties:
IntegrationId: !Ref MyIntegration
IntegrationResponseKey: /201/
ApiId: !Ref MyWebSocket
testStage:
Type: AWS::ApiGatewayV2::Stage
DependsOn:
- MyConnectRoute
- MyDefaultRoute
- MyResponseRoute
Properties:
ApiId: !Ref MyWebSocket
StageName: testStage
MyDeployment:
Type: AWS::ApiGatewayV2::Deployment
Properties:
ApiId: !Ref MyWebSocket
Description: My Deployment
StageName: !Ref testStage
The stack is build without any errors and (almost) everything looks like I intended. However other than in the manually build version the integration of the Lambda function into the websocket does not seem to add the required trigger for the Lambda function. When I manually add a Lambda function to an API Gateway, this automatically adds the trigger.
What do I need to change in my CloudFormation Yaml file to also add the trigger to the Lambda function?
Trigger automatically added to Lambda function when Lambda function is manually added to API Gateway:
No trigger added when Lambda function is added to API Gateway using CloudFormation:
It looks like you may just need to include the Lambda permission for api gateway.
LambdaFunctionPermission:
Type: "AWS::Lambda::Permission"
Properties:
Action: "lambda:InvokeFunction"
Principal: apigateway.amazonaws.com
FunctionName: !Ref MyLambdaFunction
DependsOn: MyWebSocket
If you are having issues with Cloudformation creation it would probably be helpful to also create an account config for logging and chain the dependsOn order of the gateway resources like this.
ApiGwAccountConfig:
Type: "AWS::ApiGateway::Account"
Properties:
CloudWatchRoleArn: !GetAtt "rRoleForCloudtrail.Arn"
rRoleForCloudtrail:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- apigateway.amazonaws.com
Action:
- sts:AssumeRole
Policies:
-
PolicyName: "apigateway-cloudwatch"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:DescribeLogGroups
- logs:DescribeLogStreams
- logs:PutLogEvents
- logs:GetLogEvents
- logs:FilterLogEvents
Resource: "*"
testStage:
Type: AWS::ApiGatewayV2::Stage
Properties:
ApiId: !Ref MyWebSocket
StageName: testStage
DefaultRouteSettings:
DetailedMetricsEnabled: true
LoggingLevel: INFO
DataTraceEnabled: true
DependsOn: ApiGwAccountConfig
MyDeployment:
Type: AWS::ApiGatewayV2::Deployment
DependsOn:
- MyConnectRoute
- MyDefaultRoute
- MyResponseRoute
Properties:
ApiId: !Ref MyWebSocket
Description: My Deployment
StageName: !Ref testStage
I guess you need AWS::Lambda::Permission resource, e.g.:
LambdaPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:invokeFunction
FunctionName: !GetAtt MyLambdaFunction.Arn
Principal: apigateway.amazonaws.com
SourceArn:
Fn::Join:
- ''
- - 'arn:aws:execute-api:'
- Ref: AWS::Region
- ":"
- Ref: AWS::AccountId
- ":"
- Ref: MyWebSocket
- "/*"
It may need some tweaking - it works with AWS::ApiGateway::RestApi, not sure if there is difference for AWS::ApiGatewayV2::Api.
I'm creating a pipeline with AWS Codepipeline which deploys a lambda function with a cloudformation template, here's my template:
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Parameters:
ImageUri:
Type: String
LambdaName:
Type: String
RoleName:
Type: String
DatabaseName:
Type: String
DatabasePassword:
Type: String
DatabasePort:
Type: String
DatabaseURL:
Type: String
DatabaseUsername:
Type: String
Resources:
LambdaRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Ref RoleName
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Version: 2012-10-17
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonS3FullAccess
LambdaFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Ref LambdaName
PackageType: Image
MemorySize: 256
Timeout: 5
ImageUri: !Ref ImageUri
Environment:
Variables:
DATABASE_DATABASE: !Ref DatabaseName
DATABASE_PASSWORD: !Ref DatabasePassword
DATABASE_PORT: !Ref DatabasePort
DATABASE_URL: !Ref DatabaseURL
DATABASE_USERNAME: !Ref DatabaseUsername
Role:
Fn::GetAtt:
- LambdaRole
- Arn
I'm trying to add override parmeters in codepipeline deployment stage under advanced but when I release changes it tells me that
"ParameterOverrides property is not valid".
Any help?!
I want to reuse this template but it when I up this template using nested stack it gives an error Export with name ExRole is already exported by stack Root-role. How can I improve the reuseability of the template. So that I can deploy same template in Prod, dev and other environments. I have tried using environment variable in the names of the role but how can I use it in the output and if the output is to be used in next template what should be the syntax?
Role:
---
AWSTemplateFormatVersion: 2010-09-09
Parameters:
Env:
Type: String
Resources:
ExRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
RoleName: !Sub "excutionrole-${Env}"
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
Policies:
- PolicyName: AccessECR
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- ecr:BatchGetImage
- ecr:GetAuthorizationToken
- ecr:GetDownloadUrlForLayer
Resource: '*'
ContainerInstanceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
Path: '/'
RoleName: !Sub "ContainerInstanceRole-${Env}"
InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Roles:
- !Ref ContainerInstanceRole
Outputs:
ExRole:
Description: Task excution role
Value: !Ref ExRole
Export:
Name: "ExRole"
InstanceProfile:
Description: profile for container instances
Value: !Ref InstanceProfile
Export:
Name: "InstanceProfile"
Task:
---
AWSTemplateFormatVersion: 2010-09-09
Parameters:
ExRole:
Type: String
RDS:
Type: String
DBUSER:
Type: String
Default: mysqldb
DBPASSWORD:
Type: String
Default: 1234123a
DBNAME:
Type: String
Default: mysqldb
Resources:
Task:
Type: AWS::ECS::TaskDefinition
Properties:
Family: wordpress
Cpu: 1 vCPU
ExecutionRoleArn: !Ref ExRole
Memory: 1 GB
NetworkMode: bridge
RequiresCompatibilities:
- EC2
TaskRoleArn: !Ref ExRole
ContainerDefinitions:
- Essential: true
Image: wordpress:latest
Name: wordpress
PortMappings:
- ContainerPort: 80
HostPort: 0
Protocol: tcp
Environment:
- Name: WORDPRESS_DB_HOST
Value: !Ref RDS
- Name: WORDPRESS_DB_USER
Value: !Ref DBUSER
- Name: WORDPRESS_DB_PASSWORD
Value: !Ref DBPASSWORD
- Name: WORDPRESS_DB_NAME
Value: !Ref DBNAME
Outputs:
Task:
Description: Contains all the task specifications
Value: !Ref Task
Export:
Name: "Task"
Root:
Resources:
Vpcstack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${bucketname}.s3.us-east-2.amazonaws.com${bucketpath}/vpc.yml"
Parameters:
Env: !Ref Env
Cidr: !Ref Cidr
Publicsubnet1: !Ref Publicsubnet1
Publicsubnet2: !Ref Publicsubnet2
Privatesubnet1: !Ref Privatesubnet1
Privatesubnet2: !Ref Privatesubnet2
role:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${bucketname}.s3.us-east-2.amazonaws.com${bucketpath}/role.yml"
Parameters:
Env: !Ref Env
Generally when people use output, if the template is being used multiple times within the same parent stack they will prefix the export with a variable (such as the stack name) to make it unique.
This can be done using the sub intrinsic function such as in the example below
Outputs:
ExRole:
Description: Task excution role
Value: !Ref ExRole
Export:
Name: !Sub "${AWS::StackName}-ExRole"
InstanceProfile:
Description: profile for container instances
Value: !Ref InstanceProfile
Export:
Name: !Sub "${AWS::StackName}-InstanceProfile"
Then you would need to pass in this stack ID value as a parameter into the nested stack that needs to reference this file. This would again used the sub intrinsic function to reference the export name.
To get this value in the ImportValue intrinsic function you would reference it like below, to do this you would need to pass the stack name as a parameter to the stack:
Fn::ImportValue: !Sub "${NestedStack}-ExRole"
If you call the other stack from the parent stack you can ignore exporting and instead pass the output into the next stack using the GetAtt intrinsic function instead.
Resources:
Vpcstack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${bucketname}.s3.us-east-2.amazonaws.com${bucketpath}/vpc.yml"
Parameters:
Env: !Ref Env
Cidr: !Ref Cidr
Publicsubnet1: !Ref Publicsubnet1
Publicsubnet2: !Ref Publicsubnet2
Privatesubnet1: !Ref Privatesubnet1
Privatesubnet2: !Ref Privatesubnet2
role:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${bucketname}.s3.us-east-2.amazonaws.com${bucketpath}/role.yml"
Parameters:
Env: !Ref Env
dbStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${bucketname}.s3.us-east-2.amazonaws.com${bucketpath}/db.yml"
Parameters:
Role: !GetAtt role.Outputs.ExRole
You can also use the syntax of Fn::GetAtt: [role, Outputs.ExRole] also valid syntax.
I have turned my original cloudformation stack template into multiple child templates which I then call from a master template. One of the child templates possesses the following snippet, for which I'm getting the error:
An error occurred (ValidationError) when calling the ValidateTemplate operation: Template format error: Unresolved resource dependencies [GeneralPurposeContainerRole] in the Resources block of the template
BatchResourcesStack (child stack):
---
AWSTemplateFormatVersion: '2010-09-09'
Description: batch resources stack.
Parameters:
GPCEName:
Type: String
GPCEMaxVcpus:
Type: Number
Description: Max number of VCPUs for entire cluster, there are caveats to this
GPCEMinVcpus:
Type: Number
Description: Min number of VCPUs for entire cluster, there are caveats to this
GPCEDesiredVcpus:
Type: Number
Description: Desired number of VCPUs for entire cluster, there are caveats to this
GPCEVpcId:
Type: String
GPCESubnetAZ1:
Type: String
GPCEAmi:
Type: String
GPCEInstanceTypes:
Type: CommaDelimitedList
GPCESSHKeyPair:
Type: String
StackUID:
Type: String
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Subnet:
Type: AWS::EC2::Subnet
Resources:
BatchServiceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: batch.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole
IamInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Roles:
- !Ref 'EcsInstanceRole'
EcsInstanceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2008-10-17'
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
GeneralPurposeContainerRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- sts:AssumeRole
Path: '/'
Policies:
- PolicyName: ContainerS3Access
PolicyDocument:
Statement:
- Effect: Allow
Action:
- s3:PutObject
- s3:GetObject
- s3:DeleteObject
- s3:List*
Resource:
# TODO: Make these non psychcore-specific
- arn:aws:s3:::pipeline-validation/*
- arn:aws:s3:::wgs-pipeline-vqsr-test/*
- arn:aws:s3:::test-references/*
- arn:aws:s3:::psychcore-pipelines/output/*
- arn:aws:s3:::psychcore-pipelines/validation/samples/*
- arn:aws:s3:::psychcore-data/reference_indexs/*
- arn:aws:s3:::psychcore-pipelines
- arn:aws:s3:::psychcore-pipelines
- arn:aws:s3:::psychcore-data
- arn:aws:s3:::*
- Effect: Allow
Action:
- s3:List*
- s3:ListMultipartUploadParts
- s3:AbortMultipartUpload
- s3:ListBucketMultipartUploads
Resource:
- arn:aws:s3:::pipeline-validation
- arn:aws:s3:::wgs-pipeline-vqsr-test
- arn:aws:s3:::test-references
- arn:aws:s3:::psychcore-pipelines/output/
- arn:aws:s3:::psychcore-pipelines/validation/samples/
- arn:aws:s3:::psychcore-data/reference_indexs/
- arn:aws:s3:::psychcore-pipelines
- arn:aws:s3:::psychcore-pipelines
- arn:aws:s3:::psychcore-data
- arn:aws:s3:::*
GeneralPurposeComputeEnvironment:
Type: "AWS::Batch::ComputeEnvironment"
Properties:
Type: MANAGED
ComputeEnvironmentName: !Join
- '-'
- - !Ref GPCEName
- !Ref StackUID
ComputeResources:
MinvCpus:
Ref: GPCEMinVcpus
MaxvCpus:
Ref: GPCEMaxVcpus
DesiredvCpus:
Ref: GPCEDesiredVcpus
SecurityGroupIds:
- Ref: SecurityGroup
Subnets:
- Ref: Subnet
Type: 'EC2'
ImageId:
Ref: GPCEAmi
InstanceRole:
Ref: IamInstanceProfile
InstanceTypes:
Ref: GPCEInstanceTypes
Ec2KeyPair:
Ref: GPCESSHKeyPair
Tags:
Key: Name
Value: "VariantCallingBatchComputeEnvironment"
ServiceRole:
Ref: BatchServiceRole
State: ENABLED
DependsOn:
- SecurityGroup
- Subnet
- IamInstanceProfile
- BatchServiceRole
GeneralPurposeQueue:
Type: "AWS::Batch::JobQueue"
Properties:
ComputeEnvironmentOrder:
- Order: 1
ComputeEnvironment: !Ref GeneralPurposeComputeEnvironment
Priority: 1
State: ENABLED
JobQueueName: !Join
- '-'
- - "GeneralPurposeQueue"
- !Ref StackUID
DependsOn:
- GeneralPurposeComputeEnvironment
- BatchServiceRole
Parent.yaml (contains parts relevant to above child stack):
---
AWSTemplateFormatVersion: "2010-09-09"
Description: "Master template for wgs-pipeline. Calls to other stack templates."
Parameters:
GPCEName:
Default: 'GeneralPurposeVariantCallingCE'
Type: String
GPCEMaxVcpus:
Default: 128
Type: Number
Description: Max number of VCPUs for entire cluster, there are caveats to this
GPCEMinVcpus:
Default: 0
Type: Number
Description: Min number of VCPUs for entire cluster, there are caveats to this
GPCEDesiredVcpus:
Default: 0
Type: Number
Description: Desired number of VCPUs for entire cluster, there are caveats to this
GPCEVpcId:
Type: String
GPCESubnetAZ1:
Default: 'us-east-1a'
Type: String
GPCEAmi:
Default: "ami-ce6cdfb4"
Type: String
GPCEInstanceTypes:
Default: "i3.xlarge, i3.2xlarge, i3.4xlarge, i3.8xlarge, i3.16xlarge"
Type: CommaDelimitedList
GPCESSHKeyPair:
Type: String
StackUID:
Default: "1234"
Type: String
Resources:
Subnet:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.0.0.0/24
VpcId: !Ref 'VPC'
AvailabilityZone: !Ref GPCESubnetAZ1
MapPublicIpOnLaunch: 'True'
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: EC2 Security Group for instances launched in the VPC by Batch
VpcId: !Ref 'VPC'
BatchResourcesStack:
Type: AWS::CloudFormation::Stack
Properties:
Parameters:
GPCEName:
Ref: GPCEName
GPCEMaxVcpus:
Ref: GPCEMaxVcpus
GPCEMinVcpus:
Ref: GPCEMinVcpus
GPCEDesiredVcpus:
Ref: GPCEDesiredVcpus
GPCEVpcId:
Ref: GPCEVpcId
GPCESubnetAZ1:
Ref: GPCESubnetAZ1
GPCEAmi:
Ref: GPCEAmi
GPCEInstanceTypes:
Ref: GPCEInstanceTypes
GPCESSHKeyPair:
Ref: GPCESSHKeyPair
StackUID:
Ref: StackUID
SecurityGroup:
Ref: SecurityGroup
Subnet:
Ref: Subnet
TemplateURL: https://s3.amazonaws.com/CFNTemplate/batch_resources.stack.yaml
Timeout: "100"
I don't understand what the error is specifically pointing to. I put the entire Batch-child.yaml file through a YAML validator and it passed, so it shouldn't be from a formatting/indention error per se. Also, the GeneralPurposeContainerRole resource does not get referenced to anywhere else in the template, nor even in the parent stack template.
I am trying to deploy my lambda functions using CloudFormation StackSets to multiple AWS accounts and regions. But failed because of the below error
ResourceLogicalId:OfficeHoursAutoScalingStart, ResourceType:AWS::Lambda::Function, ResourceStatusReason:Error occurred while GetObject. S3 Error Code: AuthorizationHeaderMalformed. S3 Error Message: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-southeast-1'
It seems like its a permissions thing? How do I resolve this?
My template:
AWSTemplateFormatVersion : '2010-09-09'
Description: 'Skynet. AWS Management Assistant'
Parameters:
AppName:
Type: String
Description: Prefix for resources
Default: skynet-lambda-stackset
ArtifactsBucket:
Type: String
Description: S3 bucket storing lambda function zip
ArtifactZipPath:
Type: String
Description: Path to lambda function zip
CostCenter:
Type: String
Description: Cost center
Default: Admin
Owner:
Type: String
Description: Owner
Default: Jiew Meng
Resources:
LambdaRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub '${AppName}-lambda'
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
- apigateway.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AmazonEC2FullAccess'
- 'arn:aws:iam::aws:policy/AWSLambdaFullAccess'
- 'arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess'
- 'arn:aws:iam::aws:policy/AmazonAPIGatewayInvokeFullAccess'
- 'arn:aws:iam::aws:policy/CloudWatchLogsFullAccess'
NewEc2AutoTag:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: ec2/newEc2_autoTag.handler
Runtime: nodejs6.10
FunctionName: 'NewEC2_AutoTag'
Description: 'Auto tag new EC2 instances with Owner tag'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
NewEc2Event:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${AppName}-newEc2
Description: On new EC2 instance created
EventPattern:
source:
- 'aws.ec2'
detail-type:
- 'AWS API Call via CloudTrail'
detail:
eventName:
- RunInstances
Targets:
- !Ref NewEc2AutoTag
AfterhoursEc2Shutdown:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: ec2/afterHours_shutdown.handler
Runtime: nodejs6.10
FunctionName: 'Afterhours_Shutdown'
Description: 'Shutdown instances tagged Auto Shutdown: true'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
AfterHoursEvent:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${AppName}-afterHours
Description: Triggered on weekdays 2400 SGT
ScheduleExpression: cron(0 16 ? * MON,TUE,WED,THUR,FRI *)
Targets:
- !Ref AfterhoursEc2Shutdown
- !Ref AfterhoursAutoScalingShutdown
OfficeHoursEc2Start:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: ec2/officeHours_start.handler
Runtime: nodejs6.10
FunctionName: 'OfficeHours_Start'
Description: 'Starts instances with Auto Shutdown: true'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
OfficeHoursEvent:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${AppName}-officeHours
Description: Triggered on 7AM SGT weekdays
ScheduleExpression: cron(0 23 ? * SUN,MON,TUE,WED,THU *)
Targets:
- !Ref OfficeHoursEc2Start
- !Ref OfficeHoursAutoScalingStart
StartedEc2ConfigureDns:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: ec2/started_configureDns.handler
Runtime: nodejs6.10
FunctionName: 'StartedEc2_ConfigureDns'
Description: 'When EC2 started, configure DNS if required'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
Ec2StartedEvent:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${AppName}-ec2-started
Description: Triggered on EC2 starts
EventPattern:
source:
- 'aws.ec2'
detail-type:
- 'EC2 Instance State-change Notification'
detail:
state:
- running
Targets:
- !Ref StartedEc2ConfigureDns
AfterhoursAutoScalingShutdown:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: autoscaling/afterHours_shutdown.handler
Runtime: nodejs6.10
FunctionName: 'Afterhours_AutoScalingShutdown'
Description: 'Scales down autoscaling groups tagged Auto Shutdown: true'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
OfficeHoursAutoScalingStart:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: autoscaling/officeHours_start.handler
Runtime: nodejs6.10
FunctionName: 'OfficeHours_AutoScalingStart'
Description: 'Scales up auto scaling groups that are scaled down to 0 and tagged autostart: true'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
NewAutoScalingGroupEvent:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${AppName}-autoscaling-new
Description: Triggered when new autoscaling group created
EventPattern:
source:
- 'aws.autoscaling'
detail-type:
- 'AWS API Call via CloudTrail'
detail:
eventName:
- CreateAutoScalingGroup
Targets:
- !Ref NewAutoScalingGroupAutoTag
NewAutoScalingGroupAutoTag:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: autoscaling/new_autoTag.handler
Runtime: nodejs6.10
FunctionName: 'NewAutoScalingGroup_AutoTag'
Description: 'Tags new autoscaling groups with owner and autoshutdown tags if not existing'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
Looks like you have created the s3 bucket (referenced by variable ArtifactsBucket in your template) in AWS region ap-southeast-1.
Using AWS Stacksets, You have selected us-east-1 as one of the regions in Deployment Order.
The AWS Stackset passes the SAME parameters to all the stacks which it tries to create in multiple regions/accounts.
So when it is trying to create the lambda function OfficeHoursAutoScalingStart in us-east-1 region, It is tryin to access the s3 bucket(GETObject request) in us-east-1 region itself, with the same bucket name.
ie. It is presuming that the s3 bucket with name passed by ArtifactsBucketparameter, is present in us-east-1 itself.But since the source code of the lambda function is actually in the bucket present in region ap-southeast-1,the header malformed error is thrown. In this case the bucket name is matching, but the region is not.
Currently, when you create lambda function using CloudFormation, there is a restriction that the S3 bucket that contains the source code of your Lambda function must be in the SAME region as the STACK which you are creating. Doc Reference Link
If this is the issue, then as a fix, you can think of creating s3 buckets (add region-name as a prefix to the bucket name) in the required regions and use them in the template based on the region.
Example:
us-east-1-lambdabkt
us-east-2-lambdabkt
ap-southeast-1-lambdabkt