How Do You Add Environment Variables to CodeBuild buildspec.yml? - amazon-web-services

Question

I'm trying to use a CloudFormation template to define CodeBuild and CodePipeline to automate the deployment of a static website hosted in an S3 bucket. To give credit where credit's due, I'm largely following the template from https://dzone.com/articles/continuous-delivery-to-s3-via-codepipeline-and-cod.
The problem I can't resolve is that after I add an environment variable for the Hugo version I'd like to use to create the static site files, I get an error from the AWS console that reads: "Template validation error: Template format error: Unresolved resource dependencies [HUGO_VERSION] in the Resources block of the template."
Why isn't it accepting the HUGO_VERSION environment variable that I define under environment_variables? This is version 0.1 of the format, so it's a little different than the current 0.2, but I've read the following link: https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
The thing that really confuses me is that if I remove the lines with ${HUGO_VERSION}, the template is accepted just fine - and then inspection of the CloudWatch logs after building shows (because of the printenv command) HUGO_VERSION=0.49! What gives?
Originally, the template looks like this.
---
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
SiteBucketName:
Type: String
Description: Name of bucket to create to host the website
GitHubUser:
Type: String
Description: GitHub User
Default: "stelligent"
GitHubRepo:
Type: String
Description: GitHub Repo to pull from. Only the Name. not the URL
Default: "devops-essentials"
GitHubBranch:
Type: String
Description: GitHub Branch
Default: "master"
GitHubToken:
NoEcho: true
Type: String
Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
BuildType:
Type: String
Default: "LINUX_CONTAINER"
Description: The build container type to use for building the app
BuildComputeType:
Type: String
Default: "BUILD_GENERAL1_SMALL"
Description: The build compute type to use for building the app
BuildImage:
Type: String
Default: "aws/codebuild/ubuntu-base:14.04"
Description: The build image to use for building the app
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Site Configuration"
Parameters:
- SiteBucketName
- Label:
default: "GitHub Configuration"
Parameters:
- GitHubToken
- GitHubUser
- GitHubRepo
- GitHubBranch
- Label:
default: "Build Configuration"
Parameters:
- BuildType
- BuildComputeType
- BuildImage
ParameterLabels:
SiteBucketName:
default: Name of S3 Bucket to create for website hosting
GitHubToken:
default: GitHub OAuth2 Token
GitHubUser:
default: GitHub User/Org Name
GitHubRepo:
default: GitHub Repository Name
GitHubBranch:
default: GitHub Branch Name
BuildType:
default: CodeBuild type
BuildComputeType:
default: CodeBuild instance type
BuildImage:
default: CodeBuild image
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codebuild-service
PolicyDocument:
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
Version: '2012-10-17'
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codepipeline-service
PolicyDocument:
Statement:
- Action:
- codebuild:*
Resource: "*"
Effect: Allow
- Action:
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketVersioning
Resource: "*"
Effect: Allow
- Action:
- s3:PutObject
Resource:
- arn:aws:s3:::codepipeline*
Effect: Allow
- Action:
- s3:*
- cloudformation:*
- iam:PassRole
Resource: "*"
Effect: Allow
Version: '2012-10-17'
SiteBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
AccessControl: PublicRead
BucketName: !Ref SiteBucketName
WebsiteConfiguration:
IndexDocument: index.html
PipelineBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.1
phases:
post_build:
commands:
- aws s3 cp --recursive --acl public-read ./samples s3://${SiteBucketName}/samples
- aws s3 cp --recursive --acl public-read ./html s3://${SiteBucketName}/
artifacts:
type: zip
files:
- ./html/index.html
TimeoutInMinutes: 10
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Source
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: '1'
Provider: GitHub
OutputArtifacts:
- Name: SourceArtifacts
Configuration:
Owner: !Ref GitHubUser
Repo: !Ref GitHubRepo
Branch: !Ref GitHubBranch
OAuthToken: !Ref GitHubToken
RunOrder: 1
- Name: Deploy
Actions:
- Name: Artifact
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
InputArtifacts:
- Name: SourceArtifacts
OutputArtifacts:
- Name: DeploymentArtifacts
Configuration:
ProjectName: !Ref CodeBuildDeploySite
RunOrder: 1
ArtifactStore:
Type: S3
Location: !Ref PipelineBucket
Outputs:
PipelineUrl:
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
Description: CodePipeline URL
SiteUrl:
Value: !GetAtt [SiteBucket, WebsiteURL]
Description: S3 Website URL
Now after I try to add an environment variable to use Hugo in the pipeline, the template looks like this.
---
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
SiteBucketName:
Type: String
Description: Name of bucket to create to host the website
GitHubUser:
Type: String
Description: GitHub User
Default: "stelligent"
GitHubRepo:
Type: String
Description: GitHub Repo to pull from. Only the Name. not the URL
Default: "devops-essentials"
GitHubBranch:
Type: String
Description: GitHub Branch
Default: "master"
GitHubToken:
NoEcho: true
Type: String
Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
BuildType:
Type: String
Default: "LINUX_CONTAINER"
Description: The build container type to use for building the app
BuildComputeType:
Type: String
Default: "BUILD_GENERAL1_SMALL"
Description: The build compute type to use for building the app
BuildImage:
Type: String
Default: "aws/codebuild/ubuntu-base:14.04"
Description: The build image to use for building the app
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Site Configuration"
Parameters:
- SiteBucketName
- Label:
default: "GitHub Configuration"
Parameters:
- GitHubToken
- GitHubUser
- GitHubRepo
- GitHubBranch
- Label:
default: "Build Configuration"
Parameters:
- BuildType
- BuildComputeType
- BuildImage
ParameterLabels:
SiteBucketName:
default: Name of S3 Bucket to create for website hosting
GitHubToken:
default: GitHub OAuth2 Token
GitHubUser:
default: GitHub User/Org Name
GitHubRepo:
default: GitHub Repository Name
GitHubBranch:
default: GitHub Branch Name
BuildType:
default: CodeBuild type
BuildComputeType:
default: CodeBuild instance type
BuildImage:
default: CodeBuild image
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codebuild-service
PolicyDocument:
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
Version: '2012-10-17'
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codepipeline-service
PolicyDocument:
Statement:
- Action:
- codebuild:*
Resource: "*"
Effect: Allow
- Action:
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketVersioning
Resource: "*"
Effect: Allow
- Action:
- s3:PutObject
Resource:
- arn:aws:s3:::codepipeline*
Effect: Allow
- Action:
- s3:*
- cloudformation:*
- iam:PassRole
Resource: "*"
Effect: Allow
Version: '2012-10-17'
SiteBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
AccessControl: PublicRead
BucketName: !Ref SiteBucketName
WebsiteConfiguration:
IndexDocument: index.html
PipelineBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.1
environment_variables:
plaintext:
AWS_DEFAULT_REGION: "US-WEST-2"
HUGO_VERSION: "0.49"
phases:
install:
commands:
- printenv
- echo "Install step..."
- curl -Ls https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
- tar xf /tmp/hugo.tar.gz -C /tmp
- mv /tmp/hugo_${HUGO_VERSION}_linux_amd64/hugo_${HUGO_VERSION}_linux_amd64 /usr/bin/hugo
- rm -rf /tmp/hugo*
build:
commands:
- hugo
post_build:
commands:
- aws s3 cp --recursive --acl public-read ./public s3://${SiteBucketName}
artifacts:
type: zip
files:
- ./html/index.html
TimeoutInMinutes: 10
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Source
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: '1'
Provider: GitHub
OutputArtifacts:
- Name: SourceArtifacts
Configuration:
Owner: !Ref GitHubUser
Repo: !Ref GitHubRepo
Branch: !Ref GitHubBranch
OAuthToken: !Ref GitHubToken
RunOrder: 1
- Name: Deploy
Actions:
- Name: Artifact
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
InputArtifacts:
- Name: SourceArtifacts
OutputArtifacts:
- Name: DeploymentArtifacts
Configuration:
ProjectName: !Ref CodeBuildDeploySite
RunOrder: 1
ArtifactStore:
Type: S3
Location: !Ref PipelineBucket
Outputs:
PipelineUrl:
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
Description: CodePipeline URL
SiteUrl:
Value: !GetAtt [SiteBucket, WebsiteURL]
Description: S3 Website URL
EDIT 10/20
Still haven't solved this. I tried to follow the advice given below by matsev, but I'm still getting the same validation error. For completeness, the latest template I'm trying is
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
SiteBucketName:
Type: String
Description: Name of bucket to create to host the website
GitHubUser:
Type: String
Description: GitHub User
Default: "stelligent"
GitHubRepo:
Type: String
Description: GitHub Repo to pull from. Only the Name. not the URL
Default: "devops-essentials"
GitHubBranch:
Type: String
Description: GitHub Branch
Default: "master"
GitHubToken:
NoEcho: true
Type: String
Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
BuildType:
Type: String
Default: "LINUX_CONTAINER"
Description: The build container type to use for building the app
BuildComputeType:
Type: String
Default: "BUILD_GENERAL1_SMALL"
Description: The build compute type to use for building the app
BuildImage:
Type: String
Default: "aws/codebuild/ubuntu-base:14.04"
Description: The build image to use for building the app
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Site Configuration"
Parameters:
- SiteBucketName
- Label:
default: "GitHub Configuration"
Parameters:
- GitHubToken
- GitHubUser
- GitHubRepo
- GitHubBranch
- Label:
default: "Build Configuration"
Parameters:
- BuildType
- BuildComputeType
- BuildImage
ParameterLabels:
SiteBucketName:
default: Name of S3 Bucket to create for website hosting
GitHubToken:
default: GitHub OAuth2 Token
GitHubUser:
default: GitHub User/Org Name
GitHubRepo:
default: GitHub Repository Name
GitHubBranch:
default: GitHub Branch Name
BuildType:
default: CodeBuild type
BuildComputeType:
default: CodeBuild instance type
BuildImage:
default: CodeBuild image
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codebuild-service
PolicyDocument:
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
Version: '2012-10-17'
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codepipeline-service
PolicyDocument:
Statement:
- Action:
- codebuild:*
Resource: "*"
Effect: Allow
- Action:
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketVersioning
Resource: "*"
Effect: Allow
- Action:
- s3:PutObject
Resource:
- arn:aws:s3:::codepipeline*
Effect: Allow
- Action:
- s3:*
- cloudformation:*
- iam:PassRole
Resource: "*"
Effect: Allow
Version: '2012-10-17'
SiteBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
AccessControl: PublicRead
BucketName: !Ref SiteBucketName
WebsiteConfiguration:
IndexDocument: index.html
PipelineBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
EnvironmentVariables:
- Name: HUGO_VERSION
Value: '0.49'
Type: PLAINTEXT
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.2
env:
variables:
AWS_DEFAULT_REGION: "US-WEST-2"
phases:
install:
commands:
- printenv
- curl -Ls https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
- tar xf /tmp/hugo.tar.gz -C /tmp
- mv /tmp/hugo_${HUGO_VERSION}_linux_amd64/hugo_${HUGO_VERSION}_linux_amd64 /usr/bin/hugo
- rm -rf /tmp/hugo*
build:
commands:
- hugo
post_build:
commands:
- aws s3 cp --recursive --acl public-read ./samples s3://${SiteBucketName}/samples
- aws s3 cp --recursive --acl public-read ./html s3://${SiteBucketName}/
artifacts:
type: zip
files:
- ./html/index.html
TimeoutInMinutes: 10
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Source
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: '1'
Provider: GitHub
OutputArtifacts:
- Name: SourceArtifacts
Configuration:
Owner: !Ref GitHubUser
Repo: !Ref GitHubRepo
Branch: !Ref GitHubBranch
OAuthToken: !Ref GitHubToken
RunOrder: 1
- Name: Deploy
Actions:
- Name: Artifact
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
InputArtifacts:
- Name: SourceArtifacts
OutputArtifacts:
- Name: DeploymentArtifacts
Configuration:
ProjectName: !Ref CodeBuildDeploySite
RunOrder: 1
ArtifactStore:
Type: S3
Location: !Ref PipelineBucket
Outputs:
PipelineUrl:
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
Description: CodePipeline URL
SiteUrl:
Value: !GetAtt [SiteBucket, WebsiteURL]
Description: S3 Website URL

Answer 1

Please check the Environment property of the AWS::CodeBuild::Project in your CloudFormation template. Specifically the EnvironmentVariables allows you to specify environment variables, e.g.
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
EnvironmentVariables:
- Name: HUGO_VERSION
Value: '0.49'
Type: PLAINTEXT
# More properties...
Now in you can reference the HUGO_VERSION as an environment variable in your buildspec.yml file, e.g.
pre_build:
commands:
- echo HUGO_VERSION $HUGO_VERSION

Answer 2

I believe the following happens:
CloudFormation attempts to resolve ${HUGO_VERSION} as a Parameter of Cloudformation Template as it is within the "!Sub" function.
From AWS documentation on Sub function
To write a dollar sign and curly braces (${}) literally, add an exclamation point (!) after the open curly brace, such as ${!Literal}. AWS CloudFormation resolves this text as ${Literal}.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-sub.html#w2ab1c21c28c59b7
Therefore your build spec phase should be similar to:
phases:
install:
commands:
- printenv
- echo "Install step..."
- curl -Ls https://github.com/gohugoio/hugo/releases/download/v${!HUGO_VERSION}/hugo_${!HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
- tar xf /tmp/hugo.tar.gz -C /tmp
- mv /tmp/hugo_${!HUGO_VERSION}_linux_amd64/hugo_${!HUGO_VERSION}_linux_amd64 /usr/bin/hugo
- rm -rf /tmp/hugo*
Hope this helps!