AWS CloudFormation CodeBuild: Can i use ECR image as Environment image

AWS CloudFormation CodeBuild: Can i use ECR image as Environment image - amazon-web-services

I have created the following Script where i am trying to use a custom image from ECR:
Adminrole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join
- "."
- - !Ref "AWS::StackName"
- !Ref "AWS::Region"
- "codebuild"
Path: "/"
AssumeRolePolicyDocument:
Statement:
- Action: ['sts:AssumeRole']
Effect: Allow
Principal:
Service: [codebuild.amazonaws.com]
Version: '2012-10-17'
Policies:
- PolicyName: "root"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action: "*"
Resource: "*"
ProjectTerrafom:
Type: AWS::CodeBuild::Project
Properties:
Name: !Join
- "_"
- - !Ref "AWS::StackName"
- !Ref "AWS::Region"
- "ProjectTerrafom"
Description: Terraform deployment
ServiceRole: !Ref Adminrole
Artifacts:
Type: no_artifacts
Environment:
Type: LINUX_CONTAINER
ComputeType: BUILD_GENERAL1_SMALL
Image: 111111.cer.ecr.eu-center-1.amazonaws.com/my_terraform
Source:
Location: !Ref "FullPathRepoNameTerraform"
Type: GITHUB_ENTERPRISE
TimeoutInMinutes: 10
Tags:
- Key: Project
Value: "Run Terraform From CodeBuild"
When i run the CodeBuild i get the following error:
BUILD_CONTAINER_UNABLE_TO_PULL_IMAGE: Unable to pull customer's container image.
CannotPullContainerError: Error response from daemon: pull access denied for
111111.cer.ecr.eu-center-1.amazonaws.com/my_terraform, repository does
not exist or may `enter code here`require 'docker login': denied: User: CodeBuild
Is this a permission issue or are we not allowed to use the ECR images for CodeBuild?

i have solved the issue by adding following line:
ImagePullCredentialsType: SERVICE_ROLE
The full part looks like this:
Environment:
Type: LINUX_CONTAINER
ComputeType: BUILD_GENERAL1_SMALL
Image: 111111.cer.ecr.eu-center-1.amazonaws.com/my_terraform
ImagePullCredentialsType: SERVICE_ROLE

You can use ECR images on code build but you need verification according to that error. So I suggest you crosscheck your docker credentials, confirm login then cross-check image URL

Related

Is it possible to trigger codebuild from aws cloudformation?

I'm trying to write a template that configures the whole ecs fargate server and its code pipeline.
There is no problem in all other configurations, but the image is empty because it is right after creating the ecr in cloudformation, and the create ecs service refers to the empty image and the process does not end.
So I want to push the server image to ecr with code build and then ecs service create to work, but I don't know how.
Could it be possible to trigger code build or code pipeline inside cloudformation?
If not, is there any way to do docker build & push?

Yes it can be done, I used it before to perform jobs like a database restore as part of stack creation (don't ask). What you need:
A custom resource lambda which kicks off the codebuild job. It will receive the codebuild project to start via a property passed to it in the cloudformation resource definition (assuming that the codebuild is determined at deploy time; else feel free to make the lambda know about which codebuild to run in whatever way makes the most sense to you).
The endpoint to call when the custom resource is complete. A custom resource will stay in CREATE_IN_PROGRESS state until the endpoint is called. It's been a while since I've used custom resources so don't remember where it comes from but I think it's found in the event that the custom resource lambda is invoked with.
Your codebuild job needs that endpoint and needs to be able to send a (GET or POST?) request to it, on both success and failure cases (you pass different params signifying success or failure).
So the overall sequence of steps is:
Define/reference the custom resource in your template, passing in whatever properties the lambda needs.
Deploy stack, custom resource lambda is invoked.
The custom resource status goes into CREATE_IN_PROGRESS
Lambda kicks off codebuild, passing in custom resource endpoint as a param or env var, and returns.
Codebuild starts doing its work.
Until the endpoint is invoked, the custom resource will remain as CREATE_IN_PROGRESS, and the stack create/update process will wait for it, even if it takes hours.
When codebuild has finished its work, it uses curl or similar to invoke that endpoint to signal it's complete.
Custom resource status goes to CREATE_COMPLETE (assuming you've invoked the endpoint with params saying it was successful).
Stack creation completes (or moves on to any resources that were dependent on the custom resource).

Yes, it is possible to trigger the Fargate deployment after pushing the image rather than when the CloudFormation template is run.
The trick is to set the DesiredCount property of AWS::ECS::Service to zero:
Service:
Type: AWS::ECS::Service
Properties:
Cluster: !Ref Cluster
DesiredCount: 0
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
SecurityGroups:
- !Ref SecG
Subnets: !Ref Subs
ServiceName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
TaskDefinition: !Ref TaskDefinition
That said, you can also choose to create a repo with an initial commit that will trigger the build as soon as the template is done executing. This requires you to upload the zipped source code to an S3 bucket and configure the CodeCommit repository like so:
Repo:
Type: AWS::CodeCommit::Repository
Properties:
Code:
BranchName: main
S3:
Bucket: some-bucket
Key: code.zip
RepositoryName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
RepositoryDescription: Repository
Triggers:
- Name: Trigger
CustomData: The Code Repository
DestinationArn: !Ref Topic
Branches:
- main
Events: [all]
Note that the some-bucket S3 bucket needs to contain the zipped .Dockerfile and any source code without any .git directory included.
You can see my implementation of this system and the rest of the stack below:
AWSTemplateFormatVersion: '2010-09-09'
Description: CloudFormation Stack to Trigger CodeBuild via CodePipeline
Parameters:
SecG:
Description: Single security group
Type: AWS::EC2::SecurityGroup::Id
Subs:
Description: Comma separated subnet IDs
Type: List<AWS::EC2::Subnet::Id>
ImagesFile:
Type: String
Default: images.json
Resources:
ArtifactBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
Properties:
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
Tags:
- Key: UseWithCodeDeploy
Value: true
CodeBuildServiceRole:
Type: AWS::IAM::Role
Properties:
Path: /
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: codebuild.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: !Sub 'ssm-${AWS::Region}-${AWS::StackName}'
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Action:
- ssm:GetParameters
- secretsmanager:GetSecretValue
Resource: '*'
- PolicyName: !Sub 'logs-${AWS::Region}-${AWS::StackName}'
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: '*'
- PolicyName: !Sub 'ecr-${AWS::Region}-${AWS::StackName}'
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: Allow
Action:
- ecr:BatchCheckLayerAvailability
- ecr:CompleteLayerUpload
- ecr:GetAuthorizationToken
- ecr:InitiateLayerUpload
- ecr:PutImage
- ecr:UploadLayerPart
- lightsail:*
Resource: '*'
- PolicyName: !Sub bkt-${ArtifactBucket}-${AWS::Region}
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Action:
- s3:ListBucket
- s3:GetBucketLocation
- s3:ListBucketVersions
- s3:GetBucketVersioning
Resource:
- !Sub arn:aws:s3:::${ArtifactBucket}
- arn:aws:s3:::some-bucket
- PolicyName: !Sub obj-${ArtifactBucket}-${AWS::Region}
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Action:
- s3:GetObject
- s3:PutObject
- s3:GetObjectAcl
- s3:PutObjectAcl
- s3:GetObjectTagging
- s3:PutObjectTagging
- s3:GetObjectVersion
- s3:GetObjectVersionAcl
- s3:PutObjectVersionAcl
Resource:
- !Sub arn:aws:s3:::${ArtifactBucket}/*
- arn:aws:s3:::some-bucket/*
CodeDeployServiceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Sid: '1'
Effect: Allow
Principal:
Service:
- codedeploy.us-east-1.amazonaws.com
- codedeploy.eu-west-1.amazonaws.com
Action: sts:AssumeRole
Path: /
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AWSCodeDeployRoleForECS
- arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole
- arn:aws:iam::aws:policy/service-role/AWSCodeDeployRoleForLambda
CodeDeployRolePolicies:
Type: AWS::IAM::Policy
Properties:
PolicyName: !Sub 'CDPolicy-${AWS::Region}-${AWS::StackName}'
PolicyDocument:
Statement:
- Effect: Allow
Resource:
- '*'
Action:
- ec2:Describe*
- Effect: Allow
Resource:
- '*'
Action:
- autoscaling:CompleteLifecycleAction
- autoscaling:DeleteLifecycleHook
- autoscaling:DescribeLifecycleHooks
- autoscaling:DescribeAutoScalingGroups
- autoscaling:PutLifecycleHook
- autoscaling:RecordLifecycleActionHeartbeat
Roles:
- !Ref CodeDeployServiceRole
CodePipelineServiceRole:
Type: AWS::IAM::Role
Properties:
Path: /
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: codepipeline.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: !Sub 'root-${AWS::Region}-${AWS::StackName}'
PolicyDocument:
Version: '2012-10-17'
Statement:
- Resource:
- !Sub 'arn:aws:s3:::${ArtifactBucket}/*'
- !Sub 'arn:aws:s3:::${ArtifactBucket}'
Effect: Allow
Action:
- s3:PutObject
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketAcl
- s3:GetBucketLocation
- Resource: "*"
Effect: Allow
Action:
- ecs:*
- Resource: "*"
Effect: Allow
Action:
- iam:PassRole
Condition:
StringLike:
iam:PassedToService:
- ecs-tasks.amazonaws.com
- Resource: !GetAtt Build.Arn
Effect: Allow
Action:
- codebuild:BatchGetBuilds
- codebuild:StartBuild
- codebuild:BatchGetBuildBatches
- codebuild:StartBuildBatch
- Resource: !GetAtt Repo.Arn
Effect: Allow
Action:
- codecommit:CancelUploadArchive
- codecommit:GetBranch
- codecommit:GetCommit
- codecommit:GetRepository
- codecommit:GetUploadArchiveStatus
- codecommit:UploadArchive
AmazonCloudWatchEventRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Principal:
Service:
- events.amazonaws.com
Action: sts:AssumeRole
Path: /
Policies:
-
PolicyName: cwe-pipeline-execution
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Action: codepipeline:StartPipelineExecution
Resource: !Sub arn:aws:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline}
AmazonCloudWatchEventRule:
Type: AWS::Events::Rule
Properties:
EventPattern:
source:
- aws.codecommit
detail-type:
- CodeCommit Repository State Change
resources:
- !GetAtt Repo.Arn
detail:
event:
- referenceCreated
- referenceUpdated
referenceType:
- branch
referenceName:
- main
Targets:
-
Arn: !Sub arn:aws:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline}
RoleArn: !GetAtt AmazonCloudWatchEventRole.Arn
Id: codepipeline-Pipeline
Topic:
Type: AWS::SNS::Topic
Properties:
Subscription:
- Endpoint: user#example.com
Protocol: email
TopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Sid: AllowPublish
Effect: Allow
Principal:
Service:
- 'codestar-notifications.amazonaws.com'
Action:
- 'SNS:Publish'
Resource:
- !Ref Topic
Topics:
- !Ref Topic
Repo:
Type: AWS::CodeCommit::Repository
Properties:
Code:
BranchName: main
S3:
Bucket: some-bucket
Key: code.zip
RepositoryName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
RepositoryDescription: Repository
Triggers:
- Name: Trigger
CustomData: The Code Repository
DestinationArn: !Ref Topic
Branches:
- main
Events: [all]
RepoUser:
Type: AWS::IAM::User
Properties:
Path: '/'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AWSCodeCommitPowerUser
RepoUserKey:
Type: AWS::IAM::AccessKey
Properties:
UserName:
!Ref RepoUser
Registry:
Type: AWS::ECR::Repository
Properties:
RepositoryName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
RepositoryPolicyText:
Version: '2012-10-17'
Statement:
- Sid: AllowPushPull
Effect: Allow
Principal:
AWS:
- !GetAtt CodeDeployServiceRole.Arn
Action:
- ecr:GetDownloadUrlForLayer
- ecr:BatchGetImage
- ecr:BatchCheckLayerAvailability
- ecr:PutImage
- ecr:InitiateLayerUpload
- ecr:UploadLayerPart
- ecr:CompleteLayerUpload
Build:
Type: AWS::CodeBuild::Project
Properties:
Artifacts:
Type: CODEPIPELINE
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.2
phases:
pre_build:
commands:
- echo "[`date`] PRE_BUILD"
- echo "Logging in to Amazon ECR..."
- aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ACCOUNT.dkr.ecr.$REGION.amazonaws.com
- IMAGE_URI="$ACCOUNT.dkr.ecr.$REGION.amazonaws.com/$REPO:$TAG"
build:
commands:
- echo "[`date`] BUILD"
- echo "Building Docker Image..."
- docker build -t $REPO:$TAG .
- docker tag $REPO:$TAG $IMAGE_URI
post_build:
commands:
- echo "[`date`] POST_BUILD"
- echo "Pushing Docker Image..."
- docker push $IMAGE_URI
- echo Writing image definitions file...
- printf '[{"name":"svc","imageUri":"%s"}]' $IMAGE_URI > $FILE
artifacts:
files: $FILE
Environment:
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/standard:6.0
Type: LINUX_CONTAINER
EnvironmentVariables:
- Name: REGION
Type: PLAINTEXT
Value: !Ref AWS::Region
- Name: ACCOUNT
Type: PLAINTEXT
Value: !Ref AWS::AccountId
- Name: TAG
Type: PLAINTEXT
Value: latest
- Name: REPO
Type: PLAINTEXT
Value: !Ref Registry
- Name: FILE
Type: PLAINTEXT
Value: !Ref ImagesFile
PrivilegedMode: true
Name: !Ref AWS::StackName
ServiceRole: !GetAtt CodeBuildServiceRole.Arn
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineServiceRole.Arn
ArtifactStore:
Type: S3
Location: !Ref ArtifactBucket
Stages:
- Name: Source
Actions:
- Name: Site
ActionTypeId:
Category: Source
Owner: AWS
Version: '1'
Provider: CodeCommit
Configuration:
RepositoryName: !GetAtt Repo.Name
BranchName: main
PollForSourceChanges: 'false'
InputArtifacts: []
OutputArtifacts:
- Name: SourceArtifact
RunOrder: 1
- Name: Build
Actions:
- Name: Docker
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
Configuration:
ProjectName: !Ref Build
InputArtifacts:
- Name: SourceArtifact
OutputArtifacts:
- Name: BuildArtifact
RunOrder: 1
- Name: Deploy
Actions:
- Name: Fargate
ActionTypeId:
Category: Deploy
Owner: AWS
Version: '1'
Provider: ECS
Configuration:
ClusterName: !Ref Cluster
FileName: !Ref ImagesFile
ServiceName: !GetAtt Service.Name
InputArtifacts:
- Name: BuildArtifact
RunOrder: 1
Cluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
FargateTaskExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
TaskRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- sts:AssumeRole
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
-
Name: svc
Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${Registry}:latest
PortMappings:
- ContainerPort: 8080
Cpu: 256
ExecutionRoleArn: !Ref FargateTaskExecutionRole
Memory: 512
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
RuntimePlatform:
CpuArchitecture: ARM64
OperatingSystemFamily: LINUX
TaskRoleArn: !Ref TaskRole
Service:
Type: AWS::ECS::Service
Properties:
Cluster: !Ref Cluster
DesiredCount: 0
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
SecurityGroups:
- !Ref SecG
Subnets: !Ref Subs
ServiceName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
TaskDefinition: !Ref TaskDefinition
Outputs:
ArtifactBucketName:
Description: ArtifactBucket S3 Bucket Name
Value: !Ref ArtifactBucket
ArtifactBucketSecureUrl:
Description: ArtifactBucket S3 Bucket Domain Name
Value: !Sub 'https://${ArtifactBucket.DomainName}'
ClusterName:
Value: !Ref Cluster
ServiceName:
Value: !GetAtt Service.Name
RepoUserAccessKey:
Description: S3 User Access Key
Value: !Ref RepoUserKey
RepoUserSecretKey:
Description: S3 User Secret Key
Value: !GetAtt RepoUserKey.SecretAccessKey
BuildArn:
Description: CodeBuild URL
Value: !GetAtt Build.Arn
RepoArn:
Description: CodeCommit Repository ARN
Value: !GetAtt Repo.Arn
RepoName:
Description: CodeCommit Repository NAme
Value: !GetAtt Repo.Name
RepoCloneUrlHttp:
Description: CodeCommit HTTP Clone URL
Value: !GetAtt Repo.CloneUrlHttp
RepoCloneUrlSsh:
Description: CodeCommit SSH Clone URL
Value: !GetAtt Repo.CloneUrlSsh
PipelineUrl:
Description: CodePipeline URL
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
RegistryUri:
Description: ECR Repository URI
Value: !GetAtt Registry.RepositoryUri
TopicArn:
Description: CodeCommit Notification SNS Topic ARN
Value: !Ref Topic
Hope this helps!

Cloud Formation Template: 403 while files are copied from private S3

Need Help. Trying to create Windows EC2 with Java installed using below template. However, I get 403 when artifacts are being copied from my Private S3 Bucket. Role to Access S3 and EC2 instance are created fine. What could be the issue?
Resources:
EC2S3Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: 'sts:AssumeRole'
Principal:
Service: ec2.amazonaws.com
Effect: Allow
Sid: ''
Policies:
- PolicyDocument:
Version: 2012-10-17
Statement:
- Action:
- 's3:GetObject'
Resource: "arn:aws:s3:::windowsartifcats"
Effect: Allow
PolicyName: AuthenticatedS3GetObjects
EC2S3InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref EC2S3Role
JavaSeleniumEC2Instance:
Type: "AWS::EC2::Instance"
Properties:
KeyName: "windowskeypair"
ImageId: ami-0229f7666f517b31e
InstanceType: "t2.micro"
SecurityGroups:
- "windows"
IamInstanceProfile: !Ref EC2S3InstanceProfile
UserData:
Fn::Base64: !Join
- ""
- - "<script> cfn-init.exe -v -s "
- !Ref "AWS::StackId"
- " -r JavaSeleniumEC2Instance --region "
- !Ref "AWS::Region"
- "</script>"
Metadata:
AWS::CloudFormation::Authentication:
S3AccessCreds:
Type: "S3"
buckets:
- "windowsartifcats"
roleName:
!Ref EC2S3Role
AWS::CloudFormation::Init:
config:
files:
"c:\\Install-Java-JDK.ps1":
content: !Join
- ''
- - 'Set-Location C:\\;'
- '.\\jdk-11.0.9_windows-x64_bin.exe /s'
"c:\\jdk-11.0.9_windows-x64_bin.exe":
source: "https://windowsartifcats.s3.us-east-1.amazonaws.com/jdk-11.0.9_windows-x64_bin.exe"
"c:\\IEDriverServer.exe":
source: "https://windowsartifcats.s3.us-east-1.amazonaws.com/IEDriverServer.exe"
commands:
"JavaInstall":
command: "powershell.exe -ExecutionPolicy RemoteSigned -Command c:\\Install-Java-JDK.ps1"
waitAfterCompletion: '180'
Logs from C:\cfn\log.cfn-init:
2020-12-21 06:28:15,608 [ERROR] Error encountered during build of config: Failed to retrieve https://windowsartifcats.s3.us-east-1.amazonaws.com/IEDriverServer.exe: HTTP Error 403

Your EC2S3Role is incorrect. The following
Resource: "arn:aws:s3:::windowsartifcats"
refers to bucket only, not its objects. To be able to download objects in your bucket, it should be:
Resource: "arn:aws:s3:::windowsartifcats/*"

Code pipeline getting failed at source stage when run from cloudformation

I am trying to create a code pipeline using cloud formation. My cloudformation template gets created successfully. However, the pipeline created gets stuck at the source stage itself saying the provided role does not have permission to access codecommit/s3.
The problem is that I have already given codecommit:* and s3:* policies to the pipeline role. Also I am using user permissions while running the cloud formation template and the user has admin permissions.
When I rerun the pipeline from the dashboard the pipeline runs successfully. Why is it getting failed when run automatically on getting created?
Following is the code where I am facing the problem:
Resources:
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
- events.amazonaws.com
Action: sts:AssumeRole
Path: /
CodePipelinePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: CodePipelinePolicy
PolicyDocument:
Statement:
- Action:
- iam:PassRole
Resource: "*"
Effect: Allow
Condition:
StringEqualsIfExists:
iam:PassedToService:
- cloudformation.amazonaws.com
- elasticbeanstalk.amazonaws.com
- ec2.amazonaws.com
- ecs-tasks.amazonaws.com
- Action:
- s3:*
- cloudwatch:*
- codecommit:*
- opsworks:*
- states:*
- appconfig:*
Resource: "*"
Effect: Allow
Version: '2012-10-17'
Roles:
- Ref: CodePipelineRole
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
ArtifactStore:
Type: S3
Location: !Ref S3Bucket
Name: SamplePipeline
RoleArn: !GetAtt [CodePipelineRole,Arn]
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Template
Region: !Ref AWS::Region
ActionTypeId:
Category: Source
Owner: AWS
Version: '1'
Provider: S3
OutputArtifacts:
- Name: template
Configuration:
S3Bucket: !Ref S3Bucket
PollForSourceChanges: 'false'
S3ObjectKey: !Sub ${NameOfArtifact}.zip
RunOrder: 1
....

How Do You Add Environment Variables to CodeBuild buildspec.yml?

I'm trying to use a CloudFormation template to define CodeBuild and CodePipeline to automate the deployment of a static website hosted in an S3 bucket. To give credit where credit's due, I'm largely following the template from https://dzone.com/articles/continuous-delivery-to-s3-via-codepipeline-and-cod.
The problem I can't resolve is that after I add an environment variable for the Hugo version I'd like to use to create the static site files, I get an error from the AWS console that reads: "Template validation error: Template format error: Unresolved resource dependencies [HUGO_VERSION] in the Resources block of the template."
Why isn't it accepting the HUGO_VERSION environment variable that I define under environment_variables? This is version 0.1 of the format, so it's a little different than the current 0.2, but I've read the following link: https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
The thing that really confuses me is that if I remove the lines with ${HUGO_VERSION}, the template is accepted just fine - and then inspection of the CloudWatch logs after building shows (because of the printenv command) HUGO_VERSION=0.49! What gives?
Originally, the template looks like this.
---
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
SiteBucketName:
Type: String
Description: Name of bucket to create to host the website
GitHubUser:
Type: String
Description: GitHub User
Default: "stelligent"
GitHubRepo:
Type: String
Description: GitHub Repo to pull from. Only the Name. not the URL
Default: "devops-essentials"
GitHubBranch:
Type: String
Description: GitHub Branch
Default: "master"
GitHubToken:
NoEcho: true
Type: String
Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
BuildType:
Type: String
Default: "LINUX_CONTAINER"
Description: The build container type to use for building the app
BuildComputeType:
Type: String
Default: "BUILD_GENERAL1_SMALL"
Description: The build compute type to use for building the app
BuildImage:
Type: String
Default: "aws/codebuild/ubuntu-base:14.04"
Description: The build image to use for building the app
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Site Configuration"
Parameters:
- SiteBucketName
- Label:
default: "GitHub Configuration"
Parameters:
- GitHubToken
- GitHubUser
- GitHubRepo
- GitHubBranch
- Label:
default: "Build Configuration"
Parameters:
- BuildType
- BuildComputeType
- BuildImage
ParameterLabels:
SiteBucketName:
default: Name of S3 Bucket to create for website hosting
GitHubToken:
default: GitHub OAuth2 Token
GitHubUser:
default: GitHub User/Org Name
GitHubRepo:
default: GitHub Repository Name
GitHubBranch:
default: GitHub Branch Name
BuildType:
default: CodeBuild type
BuildComputeType:
default: CodeBuild instance type
BuildImage:
default: CodeBuild image
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codebuild-service
PolicyDocument:
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
Version: '2012-10-17'
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codepipeline-service
PolicyDocument:
Statement:
- Action:
- codebuild:*
Resource: "*"
Effect: Allow
- Action:
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketVersioning
Resource: "*"
Effect: Allow
- Action:
- s3:PutObject
Resource:
- arn:aws:s3:::codepipeline*
Effect: Allow
- Action:
- s3:*
- cloudformation:*
- iam:PassRole
Resource: "*"
Effect: Allow
Version: '2012-10-17'
SiteBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
AccessControl: PublicRead
BucketName: !Ref SiteBucketName
WebsiteConfiguration:
IndexDocument: index.html
PipelineBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.1
phases:
post_build:
commands:
- aws s3 cp --recursive --acl public-read ./samples s3://${SiteBucketName}/samples
- aws s3 cp --recursive --acl public-read ./html s3://${SiteBucketName}/
artifacts:
type: zip
files:
- ./html/index.html
TimeoutInMinutes: 10
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Source
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: '1'
Provider: GitHub
OutputArtifacts:
- Name: SourceArtifacts
Configuration:
Owner: !Ref GitHubUser
Repo: !Ref GitHubRepo
Branch: !Ref GitHubBranch
OAuthToken: !Ref GitHubToken
RunOrder: 1
- Name: Deploy
Actions:
- Name: Artifact
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
InputArtifacts:
- Name: SourceArtifacts
OutputArtifacts:
- Name: DeploymentArtifacts
Configuration:
ProjectName: !Ref CodeBuildDeploySite
RunOrder: 1
ArtifactStore:
Type: S3
Location: !Ref PipelineBucket
Outputs:
PipelineUrl:
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
Description: CodePipeline URL
SiteUrl:
Value: !GetAtt [SiteBucket, WebsiteURL]
Description: S3 Website URL
Now after I try to add an environment variable to use Hugo in the pipeline, the template looks like this.
---
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
SiteBucketName:
Type: String
Description: Name of bucket to create to host the website
GitHubUser:
Type: String
Description: GitHub User
Default: "stelligent"
GitHubRepo:
Type: String
Description: GitHub Repo to pull from. Only the Name. not the URL
Default: "devops-essentials"
GitHubBranch:
Type: String
Description: GitHub Branch
Default: "master"
GitHubToken:
NoEcho: true
Type: String
Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
BuildType:
Type: String
Default: "LINUX_CONTAINER"
Description: The build container type to use for building the app
BuildComputeType:
Type: String
Default: "BUILD_GENERAL1_SMALL"
Description: The build compute type to use for building the app
BuildImage:
Type: String
Default: "aws/codebuild/ubuntu-base:14.04"
Description: The build image to use for building the app
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Site Configuration"
Parameters:
- SiteBucketName
- Label:
default: "GitHub Configuration"
Parameters:
- GitHubToken
- GitHubUser
- GitHubRepo
- GitHubBranch
- Label:
default: "Build Configuration"
Parameters:
- BuildType
- BuildComputeType
- BuildImage
ParameterLabels:
SiteBucketName:
default: Name of S3 Bucket to create for website hosting
GitHubToken:
default: GitHub OAuth2 Token
GitHubUser:
default: GitHub User/Org Name
GitHubRepo:
default: GitHub Repository Name
GitHubBranch:
default: GitHub Branch Name
BuildType:
default: CodeBuild type
BuildComputeType:
default: CodeBuild instance type
BuildImage:
default: CodeBuild image
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codebuild-service
PolicyDocument:
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
Version: '2012-10-17'
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codepipeline-service
PolicyDocument:
Statement:
- Action:
- codebuild:*
Resource: "*"
Effect: Allow
- Action:
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketVersioning
Resource: "*"
Effect: Allow
- Action:
- s3:PutObject
Resource:
- arn:aws:s3:::codepipeline*
Effect: Allow
- Action:
- s3:*
- cloudformation:*
- iam:PassRole
Resource: "*"
Effect: Allow
Version: '2012-10-17'
SiteBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
AccessControl: PublicRead
BucketName: !Ref SiteBucketName
WebsiteConfiguration:
IndexDocument: index.html
PipelineBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.1
environment_variables:
plaintext:
AWS_DEFAULT_REGION: "US-WEST-2"
HUGO_VERSION: "0.49"
phases:
install:
commands:
- printenv
- echo "Install step..."
- curl -Ls https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
- tar xf /tmp/hugo.tar.gz -C /tmp
- mv /tmp/hugo_${HUGO_VERSION}_linux_amd64/hugo_${HUGO_VERSION}_linux_amd64 /usr/bin/hugo
- rm -rf /tmp/hugo*
build:
commands:
- hugo
post_build:
commands:
- aws s3 cp --recursive --acl public-read ./public s3://${SiteBucketName}
artifacts:
type: zip
files:
- ./html/index.html
TimeoutInMinutes: 10
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Source
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: '1'
Provider: GitHub
OutputArtifacts:
- Name: SourceArtifacts
Configuration:
Owner: !Ref GitHubUser
Repo: !Ref GitHubRepo
Branch: !Ref GitHubBranch
OAuthToken: !Ref GitHubToken
RunOrder: 1
- Name: Deploy
Actions:
- Name: Artifact
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
InputArtifacts:
- Name: SourceArtifacts
OutputArtifacts:
- Name: DeploymentArtifacts
Configuration:
ProjectName: !Ref CodeBuildDeploySite
RunOrder: 1
ArtifactStore:
Type: S3
Location: !Ref PipelineBucket
Outputs:
PipelineUrl:
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
Description: CodePipeline URL
SiteUrl:
Value: !GetAtt [SiteBucket, WebsiteURL]
Description: S3 Website URL
EDIT 10/20
Still haven't solved this. I tried to follow the advice given below by matsev, but I'm still getting the same validation error. For completeness, the latest template I'm trying is
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
SiteBucketName:
Type: String
Description: Name of bucket to create to host the website
GitHubUser:
Type: String
Description: GitHub User
Default: "stelligent"
GitHubRepo:
Type: String
Description: GitHub Repo to pull from. Only the Name. not the URL
Default: "devops-essentials"
GitHubBranch:
Type: String
Description: GitHub Branch
Default: "master"
GitHubToken:
NoEcho: true
Type: String
Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
BuildType:
Type: String
Default: "LINUX_CONTAINER"
Description: The build container type to use for building the app
BuildComputeType:
Type: String
Default: "BUILD_GENERAL1_SMALL"
Description: The build compute type to use for building the app
BuildImage:
Type: String
Default: "aws/codebuild/ubuntu-base:14.04"
Description: The build image to use for building the app
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Site Configuration"
Parameters:
- SiteBucketName
- Label:
default: "GitHub Configuration"
Parameters:
- GitHubToken
- GitHubUser
- GitHubRepo
- GitHubBranch
- Label:
default: "Build Configuration"
Parameters:
- BuildType
- BuildComputeType
- BuildImage
ParameterLabels:
SiteBucketName:
default: Name of S3 Bucket to create for website hosting
GitHubToken:
default: GitHub OAuth2 Token
GitHubUser:
default: GitHub User/Org Name
GitHubRepo:
default: GitHub Repository Name
GitHubBranch:
default: GitHub Branch Name
BuildType:
default: CodeBuild type
BuildComputeType:
default: CodeBuild instance type
BuildImage:
default: CodeBuild image
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codebuild-service
PolicyDocument:
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
Version: '2012-10-17'
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codepipeline-service
PolicyDocument:
Statement:
- Action:
- codebuild:*
Resource: "*"
Effect: Allow
- Action:
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketVersioning
Resource: "*"
Effect: Allow
- Action:
- s3:PutObject
Resource:
- arn:aws:s3:::codepipeline*
Effect: Allow
- Action:
- s3:*
- cloudformation:*
- iam:PassRole
Resource: "*"
Effect: Allow
Version: '2012-10-17'
SiteBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
AccessControl: PublicRead
BucketName: !Ref SiteBucketName
WebsiteConfiguration:
IndexDocument: index.html
PipelineBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
EnvironmentVariables:
- Name: HUGO_VERSION
Value: '0.49'
Type: PLAINTEXT
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.2
env:
variables:
AWS_DEFAULT_REGION: "US-WEST-2"
phases:
install:
commands:
- printenv
- curl -Ls https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
- tar xf /tmp/hugo.tar.gz -C /tmp
- mv /tmp/hugo_${HUGO_VERSION}_linux_amd64/hugo_${HUGO_VERSION}_linux_amd64 /usr/bin/hugo
- rm -rf /tmp/hugo*
build:
commands:
- hugo
post_build:
commands:
- aws s3 cp --recursive --acl public-read ./samples s3://${SiteBucketName}/samples
- aws s3 cp --recursive --acl public-read ./html s3://${SiteBucketName}/
artifacts:
type: zip
files:
- ./html/index.html
TimeoutInMinutes: 10
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Source
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: '1'
Provider: GitHub
OutputArtifacts:
- Name: SourceArtifacts
Configuration:
Owner: !Ref GitHubUser
Repo: !Ref GitHubRepo
Branch: !Ref GitHubBranch
OAuthToken: !Ref GitHubToken
RunOrder: 1
- Name: Deploy
Actions:
- Name: Artifact
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
InputArtifacts:
- Name: SourceArtifacts
OutputArtifacts:
- Name: DeploymentArtifacts
Configuration:
ProjectName: !Ref CodeBuildDeploySite
RunOrder: 1
ArtifactStore:
Type: S3
Location: !Ref PipelineBucket
Outputs:
PipelineUrl:
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
Description: CodePipeline URL
SiteUrl:
Value: !GetAtt [SiteBucket, WebsiteURL]
Description: S3 Website URL

Please check the Environment property of the AWS::CodeBuild::Project in your CloudFormation template. Specifically the EnvironmentVariables allows you to specify environment variables, e.g.
CodeBuildDeploySite:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Sub ${AWS::StackName}-DeploySite
Description: Deploy site to S3
ServiceRole: !GetAtt CodeBuildRole.Arn
Artifacts:
Type: CODEPIPELINE
Environment:
Type: !Ref BuildType
ComputeType: !Ref BuildComputeType
Image: !Sub ${BuildImage}
EnvironmentVariables:
- Name: HUGO_VERSION
Value: '0.49'
Type: PLAINTEXT
# More properties...
Now in you can reference the HUGO_VERSION as an environment variable in your buildspec.yml file, e.g.
pre_build:
commands:
- echo HUGO_VERSION $HUGO_VERSION

I believe the following happens:
CloudFormation attempts to resolve ${HUGO_VERSION} as a Parameter of Cloudformation Template as it is within the "!Sub" function.
From AWS documentation on Sub function
To write a dollar sign and curly braces (${}) literally, add an exclamation point (!) after the open curly brace, such as ${!Literal}. AWS CloudFormation resolves this text as ${Literal}.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-sub.html#w2ab1c21c28c59b7
Therefore your build spec phase should be similar to:
phases:
install:
commands:
- printenv
- echo "Install step..."
- curl -Ls https://github.com/gohugoio/hugo/releases/download/v${!HUGO_VERSION}/hugo_${!HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
- tar xf /tmp/hugo.tar.gz -C /tmp
- mv /tmp/hugo_${!HUGO_VERSION}_linux_amd64/hugo_${!HUGO_VERSION}_linux_amd64 /usr/bin/hugo
- rm -rf /tmp/hugo*
Hope this helps!

How to handle multiple environments in CodePipeline?

I'm using code pipeline to deploy my infrastructure and I would like to be able to deploy it in different environments (dev, staging, prod,...).
I currently have a buildspec.yml file containing some "pip install" instructions and the "aws cloudformation package" command. I also created 2 pipelines, one for production and another for development pointing to 2 different branches on github. The problem I have is that since in both branches the files contains similar resources, I have name collision on the S3 buckets for example.
When using the AWS CLI and cloudformation to create or update a stack you can pass parameters using the --parameters option. I would like to do something similar in the 2 pipelines I've created.
What would be the best solution to tackle this issue?
The final goal is to automate the deployment of our infrastructure. Our infrastructure is composed of Users, KMS keys, Lamdbas (in python), Groups and Buckets.
I've created two pipelines following the tutorial: http://docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html
The first pipeline is linked to the master branch of the repo containing the code and the second one to the staging branch. My goal is to automate the deployment of the master branch in the production environment using the first pipeline and the staging branch in the staging environment using the second one.
My buildspec.yml file look like:
version: 0.1
phases:
install:
commands:
- pip install requests -t .
- pip install simplejson -t .
- pip install Image -t .
- aws cloudformation package --template-file image_processing_sam.yml --s3-bucket package-bucket --output-template-file new_image_processing_sam.yml
artifacts:
type: zip
files:
- new_image_processing_sam.yml
The image_processing_sam.yml file look like:
AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: Create a thumbnail for an image uploaded to S3
Resources:
ThumbnailFunction:
Type: "AWS::Serverless::Function"
Properties:
Role: !GetAtt LambdaExecutionRole.Arn
Handler: create_thumbnail.handler
Runtime: python2.7
Timeout: 30
Description: "A function computing the thumbnail for an image."
LambdaSecretEncryptionKey:
Type: "AWS::KMS::Key"
Properties:
Description: "A key used to encrypt secrets used in the Lambda functions"
Enabled: True
EnableKeyRotation: False
KeyPolicy:
Version: "2012-10-17"
Id: "lambda-secret-encryption-key"
Statement:
-
Sid: "Allow administration of the key"
Effect: "Allow"
Principal:
AWS: "arn:aws:iam::xxxxxxxxxxxxx:role/cloudformation-lambda-execution-role"
Action:
- "kms:Create*"
- "kms:Describe*"
- "kms:Enable*"
- "kms:List*"
- "kms:Put*"
- "kms:Update*"
- "kms:Revoke*"
- "kms:Disable*"
- "kms:Get*"
- "kms:Delete*"
- "kms:ScheduleKeyDeletion"
- "kms:CancelKeyDeletion"
Resource: "*"
-
Sid: "Allow use of the key"
Effect: "Allow"
Principal:
AWS:
- !GetAtt LambdaExecutionRole.Arn
Action:
- "kms:Encrypt"
- "kms:Decrypt"
- "kms:ReEncrypt*"
- "kms:GenerateDataKey*"
- "kms:DescribeKey"
Resource: "*"
LambdaExecutionRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: "LambdaExecutionRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
Policies:
-
PolicyName: LambdaKMS
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "kms:Decrypt"
Resource: "*"
-
Effect: "Allow"
Action:
- "lambda:InvokeFunction"
Resource: "*"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
UserGroup:
Type: "AWS::IAM::Group"
LambdaTriggerUser:
Type: "AWS::IAM::User"
Properties:
UserName: "LambdaTriggerUser"
LambdaTriggerUserKeys:
Type: "AWS::IAM::AccessKey"
Properties:
UserName:
Ref: LambdaTriggerUser
Users:
Type: "AWS::IAM::UserToGroupAddition"
Properties:
GroupName:
Ref: UserGroup
Users:
- Ref: LambdaTriggerUser
Policies:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: UserPolicy
PolicyDocument:
Statement:
-
Effect: "Allow"
Action:
- "lambda:InvokeFunction"
Resource:
- !GetAtt DispatcherFunction.Arn
Groups:
- Ref: UserGroup
PackageBucket:
Type: "AWS::S3::Bucket"
Properties:
BucketName: "package-bucket"
VersioningConfiguration:
Status: "Enabled"
Outputs:
LambdaTriggerUserAccessKey:
Value:
Ref: "LambdaTriggerUserKeys"
Description: "AWSAccessKeyId of LambdaTriggerUser"
LambdaTriggerUserSecretKey:
Value: !GetAtt LambdaTriggerUserKeys.SecretAccessKey
Description: "AWSSecretKey of LambdaTriggerUser"
I've added a deploy action in both pipelines to execute the change set computed during the beta action.
The first pipeline works like a charm and does everything I expect it to do. Each time I push code in the master branch, it's deployed.
The issue I'm facing is that when I push code in the staging branch, everything works in the pipeline until the deploy action is reached. The deploy action try to create a new stack but since it's exactly the same buildspec.yml and the image_processing_sam.yml that is processed, I reach name collision like below.
package-bucket already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
LambdaTriggerUser already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
LambdaExecutionRole already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
...
Is there a way to parameterize the buildspec.yml to be able to add a suffix to the name of the resources in the image_processing_sam.yml? Any other idea to achieve this is welcome.
Best regards.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-basic-walkthrough.html
Template configuration files are applied to the CloudFormation in the CodePipeline via a parameter file like this:
{
"Parameters" : {
"DBName" : "TestWordPressDB",
"DBPassword" : "TestDBRootPassword",
"DBRootPassword" : "TestDBRootPassword",
"DBUser" : "TestDBuser",
"KeyName" : "TestEC2KeyName"
}
}
Place these files in the root of your repo and they can be referenced in at least 2 ways.
In your CodePipeline CloudFormation:
Configuration:
ActionMode: REPLACE_ON_FAILURE
RoleArn: !GetAtt [CFNRole, Arn]
StackName: !Ref TestStackName
TemplateConfiguration: !Sub "TemplateSource::${TestStackConfig}"
TemplatePath: !Sub "TemplateSource::${TemplateFileName}"
Or in the console in the Template configuration field:
It is worth noting the config file format is different from CloudFormation via cli using
-- parameters
--parameters uses this format:
[
{
"ParameterKey": "team",
"ParameterValue": "AD-Student Life Applications"
},
{
"ParameterKey": "env",
"ParameterValue": "dev"
},
{
"ParameterKey": "dataSensitivity",
"ParameterValue": "public"
},
{
"ParameterKey": "app",
"ParameterValue": "events-list-test"
}
]
CodePipeline Cloudformation template configuration files use this format:
{
"Parameters" : {
"DBName" : "TestWordPressDB",
"DBPassword" : "TestDBRootPassword",
"DBRootPassword" : "TestDBRootPassword",
"DBUser" : "TestDBuser",
"KeyName" : "TestEC2KeyName"
}
}

Check the Eric Nord's answer. It is the one you are looking for.
I asked the question on the AWS forum as well here.
Here is the solution provided by AWS:
Hi,
If your goal is to have different bucket names for staging and master, then another option is to use CloudFormation parameters.
When editing an existing pipeline if you edit an action you can expand the "Advanced" panel and enter parameter overrides to specify a different bucket prefix for each stage. You can also enter parameters as a separate .json file in your artifact.
There's more details on doing that here: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-parameter-override-functions.html
Here's a full walk through with a different stack configuration for test and production: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-basic-walkthrough.html
Tim.
You should definitely follow the documentation provided. Here's the solution I came up with below.
Here is my own solution that I wasn't satisfied with.
I've added a script running at build time and modifying my template given the ARN of the CodeBuild agent building the project.
I've added "BRANCH_NAME" where naming collision can occur. The image_processing_sam.yml is now:
AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: Create a thumbnail for an image uploaded to S3
Resources:
ThumbnailFunction:
Type: "AWS::Serverless::Function"
Properties:
Role: !GetAtt LambdaExecutionRole.Arn
Handler: create_thumbnail.handler
Runtime: python2.7
Timeout: 30
Description: "A function computing the thumbnail for an image."
LambdaSecretEncryptionKey:
Type: "AWS::KMS::Key"
Properties:
Description: "A key used to encrypt secrets used in the Lambda functions"
Enabled: True
EnableKeyRotation: False
KeyPolicy:
Version: "2012-10-17"
Id: "lambda-secret-encryption-keyBRANCH_NAME"
Statement:
-
Sid: "Allow administration of the key"
Effect: "Allow"
Principal:
AWS: "arn:aws:iam::xxxxxxxxxxxxx:role/cloudformation-lambda-execution-role"
Action:
- "kms:Create*"
- "kms:Describe*"
- "kms:Enable*"
- "kms:List*"
- "kms:Put*"
- "kms:Update*"
- "kms:Revoke*"
- "kms:Disable*"
- "kms:Get*"
- "kms:Delete*"
- "kms:ScheduleKeyDeletion"
- "kms:CancelKeyDeletion"
Resource: "*"
-
Sid: "Allow use of the key"
Effect: "Allow"
Principal:
AWS:
- !GetAtt LambdaExecutionRole.Arn
Action:
- "kms:Encrypt"
- "kms:Decrypt"
- "kms:ReEncrypt*"
- "kms:GenerateDataKey*"
- "kms:DescribeKey"
Resource: "*"
LambdaExecutionRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: "LambdaExecutionRoleBRANCH_NAME"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
Policies:
-
PolicyName: LambdaKMSBRANCH_NAME
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "kms:Decrypt"
Resource: "*"
-
Effect: "Allow"
Action:
- "lambda:InvokeFunction"
Resource: "*"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
UserGroup:
Type: "AWS::IAM::Group"
LambdaTriggerUser:
Type: "AWS::IAM::User"
Properties:
UserName: "LambdaTriggerUserBRANCH_NAME"
LambdaTriggerUserKeys:
Type: "AWS::IAM::AccessKey"
Properties:
UserName:
Ref: LambdaTriggerUser
Users:
Type: "AWS::IAM::UserToGroupAddition"
Properties:
GroupName:
Ref: UserGroup
Users:
- Ref: LambdaTriggerUser
Policies:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: UserPolicyBRANCH_NAME
PolicyDocument:
Statement:
-
Effect: "Allow"
Action:
- "lambda:InvokeFunction"
Resource:
- !GetAtt DispatcherFunction.Arn
Groups:
- Ref: UserGroup
PackageBucket:
Type: "AWS::S3::Bucket"
Properties:
BucketName: "package-bucketBRANCH_NAME"
VersioningConfiguration:
Status: "Enabled"
Outputs:
LambdaTriggerUserAccessKey:
Value:
Ref: "LambdaTriggerUserKeys"
Description: "AWSAccessKeyId of LambdaTriggerUser"
LambdaTriggerUserSecretKey:
Value: !GetAtt LambdaTriggerUserKeys.SecretAccessKey
Description: "AWSSecretKey of LambdaTriggerUser"
The script.sh replacing the "BRANCH_NAME" in the template is:
#!/bin/bash
echo $CODEBUILD_AGENT_ENV_CODEBUILD_BUILD_ARN
if [[ "$CODEBUILD_AGENT_ENV_CODEBUILD_BUILD_ARN" == *"master"* ]]; then
sed "s/BRANCH_NAME//g" image_processing_sam.yml > generated_image_processing_sam.yml;
fi
if [[ "$CODEBUILD_AGENT_ENV_CODEBUILD_BUILD_ARN" == *"staging"* ]]; then
sed "s/BRANCH_NAME/staging/g" image_processing_sam.yml > generated_image_processing_sam.yml;
fi
The buildspec.yml is now:
version: 0.1
phases:
install:
commands:
# Install required module for python
- pip install requests -t .
- pip install simplejson -t .
- pip install Image -t .
- bash ./script.sh
# To be able to see any issue in the generated template
- cat generated_image_processing_sam.yml
# Package the generated cloudformation template in order to deploy
- aws cloudformation package --template-file generated_image_processing_sam.yml --s3-bucket piximate-package-bucket --output-template-file new_image_processing_sam.yml
artifacts:
type: zip
files:
- new_image_processing_sam.yml
I hope it can somehow help you. I would be glad if anyone can provide any improvement or documentation that could help.

We Keep Coding

c++ django amazon-web-services regex python-2.7 google-cloud-platform list unit-testing opengl ember.js