I'm trying my hand at Cloudformation nested stacks. The idea is that I create a VPC, S3 bucket, Codebuild project, and Codepipeline pipeline using Cloudformation.
My Problem: Cloudformation is saying that the following parameters (outputted by child stacks) require values:
Vpc
PrivateSubnet1
PrivateSubnet2
PrivateSubnet3
BucketName
These params should have values as the value exists when I look at a completed child stack in the console.
I'll just show the templates for the parent, s3, and codepipeline. With regards to these three templates the problem is that I am unable to use an output BucketName from S3Stack in my CodePipelineStack
My Code:
cfn-main.yaml
AWSTemplateFormatVersion: 2010-09-09
Description: root template for codepipeline poc
Parameters:
BucketName:
Type: String
VpcName:
Description: name of the vpc
Type: String
Default: sandbox
DockerUsername:
Type: String
Description: username for hub.docker
Default: seanturner026
DockerPassword:
Type: String
Description: password for hub.docker
Default: /codebuild/docker/password
Environment:
Type: String
Description: environment
AllowedValues:
- dev
- prod
Default: dev
Vpc:
Type: AWS::EC2::VPC::Id
PrivateSubnet1:
Type: AWS::EC2::Subnet::Id
PrivateSubnet2:
Type: AWS::EC2::Subnet::Id
PrivateSubnet3:
Type: AWS::EC2::Subnet::Id
GithubRepository:
Type: String
Description: github repository
Default: aws-codepipeline-poc
GithubBranch:
Type: String
Description: github branch
Default: master
GithubOwner:
Type: String
Description: github owner
Default: SeanTurner026
GithubToken:
Type: String
Description: github token for codepipeline
NoEcho: true
Resources:
VpcStack:
Type: AWS::CloudFormation::Stack
Properties:
Parameters:
VpcName: !Ref VpcName
TemplateURL: resources/vpc.yaml
S3Stack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: resources/s3.yaml
CodeBuildStack:
Type: AWS::CloudFormation::Stack
Properties:
Parameters:
Environment: !Ref Environment
DockerUsername: !Ref DockerUsername
DockerPassword: !Ref DockerPassword
Vpc: !GetAtt VpcStack.Outputs.VpcId
PrivateSubnet1: !GetAtt VpcStack.Outputs.PrivateSubnetId1
PrivateSubnet2: !GetAtt VpcStack.Outputs.PrivateSubnetId2
PrivateSubnet3: !GetAtt VpcStack.Outputs.PrivateSubnetId3
TemplateURL: resources/codebuild.yaml
CodePipelineStack:
Type: AWS::CloudFormation::Stack
Properties:
Parameters:
Environment: !Ref Environment
GithubRepository: !Ref GithubRepository
GithubBranch: !Ref GithubBranch
GithubOwner: !Ref GithubOwner
GithubToken: !Ref GithubToken
S3: !GetAtt S3Stack.Outputs.BucketName
TemplateURL: resources/codepipeline.yaml
s3.yaml
AWSTemplateFormatVersion: 2010-09-09
Description: s3 bucket for aws codepipeline poc
Resources:
S3:
Type: "AWS::S3::Bucket"
Properties:
BucketName: "aws-sean-codepipeline-poc"
Outputs:
BucketName:
Description: S3 bucket name
Value: !Ref S3
codepipeline.yaml -- Please see ArtifactStore. This is where cloudformation is seeing my parameter BucketName as value-less.
AWSTemplateFormatVersion: 2010-09-09
Description: codepipeline for aws codepipeline poc
Parameters:
BucketName:
Type: String
Environment:
Type: String
Description: environment
AllowedValues:
- dev
- prod
Default: dev
GithubRepository:
Type: String
Description: github repository
Default: aws-codepipeline-poc
GithubBranch:
Type: String
Description: github branch
Default: master
GithubOwner:
Type: String
Description: github owner
Default: SeanTurner026
GithubToken:
Type: String
Description: github token for codepipeline
NoEcho: true
Resources:
CodePipelineRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: !Join
- ""
- - !Ref AWS::StackName
- "-code-pipeline-role-"
- !Ref Environment
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
Effect: "Allow"
Principal:
Service: "codepipeline.amazonaws.com"
Action: "sts:AssumeRole"
CodePipelinePolicy:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: !Join
- ""
- - !Ref AWS::StackName
- "-code-pipeline-policy-"
- !Ref Environment
PolicyDocument:
Version: "2012-10-17"
Statement:
Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
- s3:putObject
- s3:getObject
- codebuild:*
Resource:
- "*"
Roles:
- !Ref CodePipelineRole
Pipeline:
Type: "AWS::CodePipeline::Pipeline"
Properties:
Name: !Join
- ""
- - "code-pipeline-poc-"
- !Ref AWS::StackName
ArtifactStore:
Location: !Ref BucketName
Type: S3
RestartExecutionOnUpdate: true
RoleArn: !Join
- ""
- - "arn:aws:iam::"
- !Ref AWS::AccountId
- ":role/"
- !Ref CodePipelineRole
Stages:
- Name: checkout-source-code
Actions:
- Name: SourceAction
RunOrder: 1
ActionTypeId:
Category: Source
Owner: ThirdParty
Provider: GitHub
Version: 1
Configuration:
Owner: !Ref GithubOwner
Repo: !Ref GithubRepository
Branch: !Ref GithubBranch
PollForSourceChanges: true
OAuthToken: !Ref GithubToken
OutputArtifacts:
- Name: source-code
- Name: docker-build-push
Actions:
- Name: build-push-job
RunOrder: 1
InputArtifacts:
- Name: source-code
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref BuildPushJob
OutputArtifacts:
- Name: build-push-job
Sorry if this is too verbose. If missed above, the problem is that ArtifactStore in the codepipeline.yaml is seeing my parameter BucketName as value-less, despite the value being outputted by S3Stack.
You pass the parameter as S3 but the template is expecting it as BucketName.
Related
I'm creating a pipeline with AWS Codepipeline which deploys a lambda function with a cloudformation template, here's my template:
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Parameters:
ImageUri:
Type: String
LambdaName:
Type: String
RoleName:
Type: String
DatabaseName:
Type: String
DatabasePassword:
Type: String
DatabasePort:
Type: String
DatabaseURL:
Type: String
DatabaseUsername:
Type: String
Resources:
LambdaRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Ref RoleName
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Version: 2012-10-17
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonS3FullAccess
LambdaFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Ref LambdaName
PackageType: Image
MemorySize: 256
Timeout: 5
ImageUri: !Ref ImageUri
Environment:
Variables:
DATABASE_DATABASE: !Ref DatabaseName
DATABASE_PASSWORD: !Ref DatabasePassword
DATABASE_PORT: !Ref DatabasePort
DATABASE_URL: !Ref DatabaseURL
DATABASE_USERNAME: !Ref DatabaseUsername
Role:
Fn::GetAtt:
- LambdaRole
- Arn
I'm trying to add override parmeters in codepipeline deployment stage under advanced but when I release changes it tells me that
"ParameterOverrides property is not valid".
Any help?!
I want to reuse this template but it when I up this template using nested stack it gives an error Export with name ExRole is already exported by stack Root-role. How can I improve the reuseability of the template. So that I can deploy same template in Prod, dev and other environments. I have tried using environment variable in the names of the role but how can I use it in the output and if the output is to be used in next template what should be the syntax?
Role:
---
AWSTemplateFormatVersion: 2010-09-09
Parameters:
Env:
Type: String
Resources:
ExRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
RoleName: !Sub "excutionrole-${Env}"
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
Policies:
- PolicyName: AccessECR
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- ecr:BatchGetImage
- ecr:GetAuthorizationToken
- ecr:GetDownloadUrlForLayer
Resource: '*'
ContainerInstanceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
Path: '/'
RoleName: !Sub "ContainerInstanceRole-${Env}"
InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Roles:
- !Ref ContainerInstanceRole
Outputs:
ExRole:
Description: Task excution role
Value: !Ref ExRole
Export:
Name: "ExRole"
InstanceProfile:
Description: profile for container instances
Value: !Ref InstanceProfile
Export:
Name: "InstanceProfile"
Task:
---
AWSTemplateFormatVersion: 2010-09-09
Parameters:
ExRole:
Type: String
RDS:
Type: String
DBUSER:
Type: String
Default: mysqldb
DBPASSWORD:
Type: String
Default: 1234123a
DBNAME:
Type: String
Default: mysqldb
Resources:
Task:
Type: AWS::ECS::TaskDefinition
Properties:
Family: wordpress
Cpu: 1 vCPU
ExecutionRoleArn: !Ref ExRole
Memory: 1 GB
NetworkMode: bridge
RequiresCompatibilities:
- EC2
TaskRoleArn: !Ref ExRole
ContainerDefinitions:
- Essential: true
Image: wordpress:latest
Name: wordpress
PortMappings:
- ContainerPort: 80
HostPort: 0
Protocol: tcp
Environment:
- Name: WORDPRESS_DB_HOST
Value: !Ref RDS
- Name: WORDPRESS_DB_USER
Value: !Ref DBUSER
- Name: WORDPRESS_DB_PASSWORD
Value: !Ref DBPASSWORD
- Name: WORDPRESS_DB_NAME
Value: !Ref DBNAME
Outputs:
Task:
Description: Contains all the task specifications
Value: !Ref Task
Export:
Name: "Task"
Root:
Resources:
Vpcstack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${bucketname}.s3.us-east-2.amazonaws.com${bucketpath}/vpc.yml"
Parameters:
Env: !Ref Env
Cidr: !Ref Cidr
Publicsubnet1: !Ref Publicsubnet1
Publicsubnet2: !Ref Publicsubnet2
Privatesubnet1: !Ref Privatesubnet1
Privatesubnet2: !Ref Privatesubnet2
role:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${bucketname}.s3.us-east-2.amazonaws.com${bucketpath}/role.yml"
Parameters:
Env: !Ref Env
Generally when people use output, if the template is being used multiple times within the same parent stack they will prefix the export with a variable (such as the stack name) to make it unique.
This can be done using the sub intrinsic function such as in the example below
Outputs:
ExRole:
Description: Task excution role
Value: !Ref ExRole
Export:
Name: !Sub "${AWS::StackName}-ExRole"
InstanceProfile:
Description: profile for container instances
Value: !Ref InstanceProfile
Export:
Name: !Sub "${AWS::StackName}-InstanceProfile"
Then you would need to pass in this stack ID value as a parameter into the nested stack that needs to reference this file. This would again used the sub intrinsic function to reference the export name.
To get this value in the ImportValue intrinsic function you would reference it like below, to do this you would need to pass the stack name as a parameter to the stack:
Fn::ImportValue: !Sub "${NestedStack}-ExRole"
If you call the other stack from the parent stack you can ignore exporting and instead pass the output into the next stack using the GetAtt intrinsic function instead.
Resources:
Vpcstack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${bucketname}.s3.us-east-2.amazonaws.com${bucketpath}/vpc.yml"
Parameters:
Env: !Ref Env
Cidr: !Ref Cidr
Publicsubnet1: !Ref Publicsubnet1
Publicsubnet2: !Ref Publicsubnet2
Privatesubnet1: !Ref Privatesubnet1
Privatesubnet2: !Ref Privatesubnet2
role:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${bucketname}.s3.us-east-2.amazonaws.com${bucketpath}/role.yml"
Parameters:
Env: !Ref Env
dbStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${bucketname}.s3.us-east-2.amazonaws.com${bucketpath}/db.yml"
Parameters:
Role: !GetAtt role.Outputs.ExRole
You can also use the syntax of Fn::GetAtt: [role, Outputs.ExRole] also valid syntax.
I'm trying to use a CloudFormation template to define CodeBuild and CodePipeline to automate the deployment of a static website hosted in an S3 bucket. To give credit where credit's due, I'm largely following the template from https://dzone.com/articles/continuous-delivery-to-s3-via-codepipeline-and-cod.
The problem I can't resolve is that after I add an environment variable for the Hugo version I'd like to use to create the static site files, I get an error from the AWS console that reads: "Template validation error: Template format error: Unresolved resource dependencies [HUGO_VERSION] in the Resources block of the template."
Why isn't it accepting the HUGO_VERSION environment variable that I define under environment_variables? This is version 0.1 of the format, so it's a little different than the current 0.2, but I've read the following link: https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
The thing that really confuses me is that if I remove the lines with ${HUGO_VERSION}, the template is accepted just fine - and then inspection of the CloudWatch logs after building shows (because of the printenv command) HUGO_VERSION=0.49! What gives?
Originally, the template looks like this.
---
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
SiteBucketName:
Type: String
Description: Name of bucket to create to host the website
GitHubUser:
Type: String
Description: GitHub User
Default: "stelligent"
GitHubRepo:
Type: String
Description: GitHub Repo to pull from. Only the Name. not the URL
Default: "devops-essentials"
GitHubBranch:
Type: String
Description: GitHub Branch
Default: "master"
GitHubToken:
NoEcho: true
Type: String
Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
BuildType:
Type: String
Default: "LINUX_CONTAINER"
Description: The build container type to use for building the app
BuildComputeType:
Type: String
Default: "BUILD_GENERAL1_SMALL"
Description: The build compute type to use for building the app
BuildImage:
Type: String
Default: "aws/codebuild/ubuntu-base:14.04"
Description: The build image to use for building the app
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Site Configuration"
Parameters:
- SiteBucketName
- Label:
default: "GitHub Configuration"
Parameters:
- GitHubToken
- GitHubUser
- GitHubRepo
- GitHubBranch
- Label:
default: "Build Configuration"
Parameters:
- BuildType
- BuildComputeType
- BuildImage
ParameterLabels:
SiteBucketName:
default: Name of S3 Bucket to create for website hosting
GitHubToken:
default: GitHub OAuth2 Token
GitHubUser:
default: GitHub User/Org Name
GitHubRepo:
default: GitHub Repository Name
GitHubBranch:
default: GitHub Branch Name
BuildType:
default: CodeBuild type
BuildComputeType:
default: CodeBuild instance type
BuildImage:
default: CodeBuild image
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codebuild-service
PolicyDocument:
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
Version: '2012-10-17'
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codepipeline-service
PolicyDocument:
Statement:
- Action:
- codebuild:*
Resource: "*"
Effect: Allow
- Action:
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketVersioning
Resource: "*"
Effect: Allow
- Action:
- s3:PutObject
Resource:
- arn:aws:s3:::codepipeline*
Effect: Allow
- Action:
- s3:*
- cloudformation:*
- iam:PassRole
Resource: "*"
Effect: Allow
Version: '2012-10-17'
SiteBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
AccessControl: PublicRead
BucketName: !Ref SiteBucketName
WebsiteConfiguration:
IndexDocument: index.html
PipelineBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.1
phases:
post_build:
commands:
- aws s3 cp --recursive --acl public-read ./samples s3://${SiteBucketName}/samples
- aws s3 cp --recursive --acl public-read ./html s3://${SiteBucketName}/
artifacts:
type: zip
files:
- ./html/index.html
TimeoutInMinutes: 10
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Source
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: '1'
Provider: GitHub
OutputArtifacts:
- Name: SourceArtifacts
Configuration:
Owner: !Ref GitHubUser
Repo: !Ref GitHubRepo
Branch: !Ref GitHubBranch
OAuthToken: !Ref GitHubToken
RunOrder: 1
- Name: Deploy
Actions:
- Name: Artifact
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
InputArtifacts:
- Name: SourceArtifacts
OutputArtifacts:
- Name: DeploymentArtifacts
Configuration:
ProjectName: !Ref CodeBuildDeploySite
RunOrder: 1
ArtifactStore:
Type: S3
Location: !Ref PipelineBucket
Outputs:
PipelineUrl:
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
Description: CodePipeline URL
SiteUrl:
Value: !GetAtt [SiteBucket, WebsiteURL]
Description: S3 Website URL
Now after I try to add an environment variable to use Hugo in the pipeline, the template looks like this.
---
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
SiteBucketName:
Type: String
Description: Name of bucket to create to host the website
GitHubUser:
Type: String
Description: GitHub User
Default: "stelligent"
GitHubRepo:
Type: String
Description: GitHub Repo to pull from. Only the Name. not the URL
Default: "devops-essentials"
GitHubBranch:
Type: String
Description: GitHub Branch
Default: "master"
GitHubToken:
NoEcho: true
Type: String
Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
BuildType:
Type: String
Default: "LINUX_CONTAINER"
Description: The build container type to use for building the app
BuildComputeType:
Type: String
Default: "BUILD_GENERAL1_SMALL"
Description: The build compute type to use for building the app
BuildImage:
Type: String
Default: "aws/codebuild/ubuntu-base:14.04"
Description: The build image to use for building the app
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Site Configuration"
Parameters:
- SiteBucketName
- Label:
default: "GitHub Configuration"
Parameters:
- GitHubToken
- GitHubUser
- GitHubRepo
- GitHubBranch
- Label:
default: "Build Configuration"
Parameters:
- BuildType
- BuildComputeType
- BuildImage
ParameterLabels:
SiteBucketName:
default: Name of S3 Bucket to create for website hosting
GitHubToken:
default: GitHub OAuth2 Token
GitHubUser:
default: GitHub User/Org Name
GitHubRepo:
default: GitHub Repository Name
GitHubBranch:
default: GitHub Branch Name
BuildType:
default: CodeBuild type
BuildComputeType:
default: CodeBuild instance type
BuildImage:
default: CodeBuild image
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codebuild-service
PolicyDocument:
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
Version: '2012-10-17'
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codepipeline-service
PolicyDocument:
Statement:
- Action:
- codebuild:*
Resource: "*"
Effect: Allow
- Action:
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketVersioning
Resource: "*"
Effect: Allow
- Action:
- s3:PutObject
Resource:
- arn:aws:s3:::codepipeline*
Effect: Allow
- Action:
- s3:*
- cloudformation:*
- iam:PassRole
Resource: "*"
Effect: Allow
Version: '2012-10-17'
SiteBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
AccessControl: PublicRead
BucketName: !Ref SiteBucketName
WebsiteConfiguration:
IndexDocument: index.html
PipelineBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.1
environment_variables:
plaintext:
AWS_DEFAULT_REGION: "US-WEST-2"
HUGO_VERSION: "0.49"
phases:
install:
commands:
- printenv
- echo "Install step..."
- curl -Ls https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
- tar xf /tmp/hugo.tar.gz -C /tmp
- mv /tmp/hugo_${HUGO_VERSION}_linux_amd64/hugo_${HUGO_VERSION}_linux_amd64 /usr/bin/hugo
- rm -rf /tmp/hugo*
build:
commands:
- hugo
post_build:
commands:
- aws s3 cp --recursive --acl public-read ./public s3://${SiteBucketName}
artifacts:
type: zip
files:
- ./html/index.html
TimeoutInMinutes: 10
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Source
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: '1'
Provider: GitHub
OutputArtifacts:
- Name: SourceArtifacts
Configuration:
Owner: !Ref GitHubUser
Repo: !Ref GitHubRepo
Branch: !Ref GitHubBranch
OAuthToken: !Ref GitHubToken
RunOrder: 1
- Name: Deploy
Actions:
- Name: Artifact
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
InputArtifacts:
- Name: SourceArtifacts
OutputArtifacts:
- Name: DeploymentArtifacts
Configuration:
ProjectName: !Ref CodeBuildDeploySite
RunOrder: 1
ArtifactStore:
Type: S3
Location: !Ref PipelineBucket
Outputs:
PipelineUrl:
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
Description: CodePipeline URL
SiteUrl:
Value: !GetAtt [SiteBucket, WebsiteURL]
Description: S3 Website URL
EDIT 10/20
Still haven't solved this. I tried to follow the advice given below by matsev, but I'm still getting the same validation error. For completeness, the latest template I'm trying is
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
SiteBucketName:
Type: String
Description: Name of bucket to create to host the website
GitHubUser:
Type: String
Description: GitHub User
Default: "stelligent"
GitHubRepo:
Type: String
Description: GitHub Repo to pull from. Only the Name. not the URL
Default: "devops-essentials"
GitHubBranch:
Type: String
Description: GitHub Branch
Default: "master"
GitHubToken:
NoEcho: true
Type: String
Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
BuildType:
Type: String
Default: "LINUX_CONTAINER"
Description: The build container type to use for building the app
BuildComputeType:
Type: String
Default: "BUILD_GENERAL1_SMALL"
Description: The build compute type to use for building the app
BuildImage:
Type: String
Default: "aws/codebuild/ubuntu-base:14.04"
Description: The build image to use for building the app
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Site Configuration"
Parameters:
- SiteBucketName
- Label:
default: "GitHub Configuration"
Parameters:
- GitHubToken
- GitHubUser
- GitHubRepo
- GitHubBranch
- Label:
default: "Build Configuration"
Parameters:
- BuildType
- BuildComputeType
- BuildImage
ParameterLabels:
SiteBucketName:
default: Name of S3 Bucket to create for website hosting
GitHubToken:
default: GitHub OAuth2 Token
GitHubUser:
default: GitHub User/Org Name
GitHubRepo:
default: GitHub Repository Name
GitHubBranch:
default: GitHub Branch Name
BuildType:
default: CodeBuild type
BuildComputeType:
default: CodeBuild instance type
BuildImage:
default: CodeBuild image
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codebuild-service
PolicyDocument:
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
Version: '2012-10-17'
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codepipeline-service
PolicyDocument:
Statement:
- Action:
- codebuild:*
Resource: "*"
Effect: Allow
- Action:
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketVersioning
Resource: "*"
Effect: Allow
- Action:
- s3:PutObject
Resource:
- arn:aws:s3:::codepipeline*
Effect: Allow
- Action:
- s3:*
- cloudformation:*
- iam:PassRole
Resource: "*"
Effect: Allow
Version: '2012-10-17'
SiteBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
AccessControl: PublicRead
BucketName: !Ref SiteBucketName
WebsiteConfiguration:
IndexDocument: index.html
PipelineBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
EnvironmentVariables:
- Name: HUGO_VERSION
Value: '0.49'
Type: PLAINTEXT
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.2
env:
variables:
AWS_DEFAULT_REGION: "US-WEST-2"
phases:
install:
commands:
- printenv
- curl -Ls https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
- tar xf /tmp/hugo.tar.gz -C /tmp
- mv /tmp/hugo_${HUGO_VERSION}_linux_amd64/hugo_${HUGO_VERSION}_linux_amd64 /usr/bin/hugo
- rm -rf /tmp/hugo*
build:
commands:
- hugo
post_build:
commands:
- aws s3 cp --recursive --acl public-read ./samples s3://${SiteBucketName}/samples
- aws s3 cp --recursive --acl public-read ./html s3://${SiteBucketName}/
artifacts:
type: zip
files:
- ./html/index.html
TimeoutInMinutes: 10
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Source
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: '1'
Provider: GitHub
OutputArtifacts:
- Name: SourceArtifacts
Configuration:
Owner: !Ref GitHubUser
Repo: !Ref GitHubRepo
Branch: !Ref GitHubBranch
OAuthToken: !Ref GitHubToken
RunOrder: 1
- Name: Deploy
Actions:
- Name: Artifact
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
InputArtifacts:
- Name: SourceArtifacts
OutputArtifacts:
- Name: DeploymentArtifacts
Configuration:
ProjectName: !Ref CodeBuildDeploySite
RunOrder: 1
ArtifactStore:
Type: S3
Location: !Ref PipelineBucket
Outputs:
PipelineUrl:
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
Description: CodePipeline URL
SiteUrl:
Value: !GetAtt [SiteBucket, WebsiteURL]
Description: S3 Website URL
Please check the Environment property of the AWS::CodeBuild::Project in your CloudFormation template. Specifically the EnvironmentVariables allows you to specify environment variables, e.g.
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
EnvironmentVariables:
- Name: HUGO_VERSION
Value: '0.49'
Type: PLAINTEXT
# More properties...
Now in you can reference the HUGO_VERSION as an environment variable in your buildspec.yml file, e.g.
pre_build:
commands:
- echo HUGO_VERSION $HUGO_VERSION
I believe the following happens:
CloudFormation attempts to resolve ${HUGO_VERSION} as a Parameter of Cloudformation Template as it is within the "!Sub" function.
From AWS documentation on Sub function
To write a dollar sign and curly braces (${}) literally, add an exclamation point (!) after the open curly brace, such as ${!Literal}. AWS CloudFormation resolves this text as ${Literal}.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-sub.html#w2ab1c21c28c59b7
Therefore your build spec phase should be similar to:
phases:
install:
commands:
- printenv
- echo "Install step..."
- curl -Ls https://github.com/gohugoio/hugo/releases/download/v${!HUGO_VERSION}/hugo_${!HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
- tar xf /tmp/hugo.tar.gz -C /tmp
- mv /tmp/hugo_${!HUGO_VERSION}_linux_amd64/hugo_${!HUGO_VERSION}_linux_amd64 /usr/bin/hugo
- rm -rf /tmp/hugo*
Hope this helps!
I am trying to deploy my lambda functions using CloudFormation StackSets to multiple AWS accounts and regions. But failed because of the below error
ResourceLogicalId:OfficeHoursAutoScalingStart, ResourceType:AWS::Lambda::Function, ResourceStatusReason:Error occurred while GetObject. S3 Error Code: AuthorizationHeaderMalformed. S3 Error Message: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-southeast-1'
It seems like its a permissions thing? How do I resolve this?
My template:
AWSTemplateFormatVersion : '2010-09-09'
Description: 'Skynet. AWS Management Assistant'
Parameters:
AppName:
Type: String
Description: Prefix for resources
Default: skynet-lambda-stackset
ArtifactsBucket:
Type: String
Description: S3 bucket storing lambda function zip
ArtifactZipPath:
Type: String
Description: Path to lambda function zip
CostCenter:
Type: String
Description: Cost center
Default: Admin
Owner:
Type: String
Description: Owner
Default: Jiew Meng
Resources:
LambdaRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub '${AppName}-lambda'
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
- apigateway.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AmazonEC2FullAccess'
- 'arn:aws:iam::aws:policy/AWSLambdaFullAccess'
- 'arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess'
- 'arn:aws:iam::aws:policy/AmazonAPIGatewayInvokeFullAccess'
- 'arn:aws:iam::aws:policy/CloudWatchLogsFullAccess'
NewEc2AutoTag:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: ec2/newEc2_autoTag.handler
Runtime: nodejs6.10
FunctionName: 'NewEC2_AutoTag'
Description: 'Auto tag new EC2 instances with Owner tag'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
NewEc2Event:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${AppName}-newEc2
Description: On new EC2 instance created
EventPattern:
source:
- 'aws.ec2'
detail-type:
- 'AWS API Call via CloudTrail'
detail:
eventName:
- RunInstances
Targets:
- !Ref NewEc2AutoTag
AfterhoursEc2Shutdown:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: ec2/afterHours_shutdown.handler
Runtime: nodejs6.10
FunctionName: 'Afterhours_Shutdown'
Description: 'Shutdown instances tagged Auto Shutdown: true'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
AfterHoursEvent:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${AppName}-afterHours
Description: Triggered on weekdays 2400 SGT
ScheduleExpression: cron(0 16 ? * MON,TUE,WED,THUR,FRI *)
Targets:
- !Ref AfterhoursEc2Shutdown
- !Ref AfterhoursAutoScalingShutdown
OfficeHoursEc2Start:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: ec2/officeHours_start.handler
Runtime: nodejs6.10
FunctionName: 'OfficeHours_Start'
Description: 'Starts instances with Auto Shutdown: true'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
OfficeHoursEvent:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${AppName}-officeHours
Description: Triggered on 7AM SGT weekdays
ScheduleExpression: cron(0 23 ? * SUN,MON,TUE,WED,THU *)
Targets:
- !Ref OfficeHoursEc2Start
- !Ref OfficeHoursAutoScalingStart
StartedEc2ConfigureDns:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: ec2/started_configureDns.handler
Runtime: nodejs6.10
FunctionName: 'StartedEc2_ConfigureDns'
Description: 'When EC2 started, configure DNS if required'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
Ec2StartedEvent:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${AppName}-ec2-started
Description: Triggered on EC2 starts
EventPattern:
source:
- 'aws.ec2'
detail-type:
- 'EC2 Instance State-change Notification'
detail:
state:
- running
Targets:
- !Ref StartedEc2ConfigureDns
AfterhoursAutoScalingShutdown:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: autoscaling/afterHours_shutdown.handler
Runtime: nodejs6.10
FunctionName: 'Afterhours_AutoScalingShutdown'
Description: 'Scales down autoscaling groups tagged Auto Shutdown: true'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
OfficeHoursAutoScalingStart:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: autoscaling/officeHours_start.handler
Runtime: nodejs6.10
FunctionName: 'OfficeHours_AutoScalingStart'
Description: 'Scales up auto scaling groups that are scaled down to 0 and tagged autostart: true'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
NewAutoScalingGroupEvent:
Type: AWS::Events::Rule
Properties:
Name: !Sub ${AppName}-autoscaling-new
Description: Triggered when new autoscaling group created
EventPattern:
source:
- 'aws.autoscaling'
detail-type:
- 'AWS API Call via CloudTrail'
detail:
eventName:
- CreateAutoScalingGroup
Targets:
- !Ref NewAutoScalingGroupAutoTag
NewAutoScalingGroupAutoTag:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref ArtifactZipPath
Handler: autoscaling/new_autoTag.handler
Runtime: nodejs6.10
FunctionName: 'NewAutoScalingGroup_AutoTag'
Description: 'Tags new autoscaling groups with owner and autoshutdown tags if not existing'
Timeout: 30
Role: !GetAtt LambdaRole.Arn
Tags:
- Key: Cost Center
Value: !Ref CostCenter
- Key: Owner
Value: !Ref Owner
Looks like you have created the s3 bucket (referenced by variable ArtifactsBucket in your template) in AWS region ap-southeast-1.
Using AWS Stacksets, You have selected us-east-1 as one of the regions in Deployment Order.
The AWS Stackset passes the SAME parameters to all the stacks which it tries to create in multiple regions/accounts.
So when it is trying to create the lambda function OfficeHoursAutoScalingStart in us-east-1 region, It is tryin to access the s3 bucket(GETObject request) in us-east-1 region itself, with the same bucket name.
ie. It is presuming that the s3 bucket with name passed by ArtifactsBucketparameter, is present in us-east-1 itself.But since the source code of the lambda function is actually in the bucket present in region ap-southeast-1,the header malformed error is thrown. In this case the bucket name is matching, but the region is not.
Currently, when you create lambda function using CloudFormation, there is a restriction that the S3 bucket that contains the source code of your Lambda function must be in the SAME region as the STACK which you are creating. Doc Reference Link
If this is the issue, then as a fix, you can think of creating s3 buckets (add region-name as a prefix to the bucket name) in the required regions and use them in the template based on the region.
Example:
us-east-1-lambdabkt
us-east-2-lambdabkt
ap-southeast-1-lambdabkt
I have an SAM template that creates an API Gateway API exporting the API Endpoint:
AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: 'API Server for Skynet: AWS Management Assistant'
Resources:
ApiGatewayApi:
Type: AWS::Serverless::Api
Properties:
DefinitionUri: swagger.yml
StageName: prod
Variables:
Region: !Ref AWS::Region
AccountId: !Ref AWS::AccountId
Outputs:
ApiEndpoint:
Description: API Endpoint
Value: !Sub
- 'https://${ApiId}.execute-api.${AWS::Region}.amazonaws.com/prod'
- {ApiId: !Ref ApiGatewayApi}
Export:
Name: !Sub '${AWS::StackName}-ApiEndpoint'
In my CodePipeline/CodeBuild stack: I am referencing it:
CodeBuildWeb:
Type: AWS::CodeBuild::Project
Properties:
Name: !Sub '${PipelineName}-web'
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/nodejs:7.0.0
Type: LINUX_CONTAINER
EnvironmentVariables:
- Name: S3_BUCKET
Value: !Ref S3WebBucket
- Name: API_URL
Value: Fn::ImportValue
!Sub '${PipelineName}-server-ApiEndpoint'
ServiceRole: !Ref CodeBuildRole
Source:
BuildSpec: 'web/buildspec.yml'
Type: CODEPIPELINE
Problem is, its not expanding, in my code, I see ApiEndpoint as Fn::ImportValue !Sub '${PipelineName}-server-ApiEndpoint' whats wrong here?
OK, I found my mistake:
- Name: API_URL
Value: Fn::ImportValue:
!Sub '${PipelineName}-server-ApiEndpoint'
I should add a : after Fn::ImportValue