aws CodePipeline manual approval gets skipped - amazon-web-services

Question

We have pipelines that pull the code from a CodeCommit repository and builds and deploys the code.
Between the Source stage and the Build stage, there is a manual approval stage and for random reason this stage sometimes gets skipped and the Pipeline continues directly to the Build stage without getting approved or rejected. Some other times even when it's approved, the manual approval stage would remain pending and waiting for approval. Some other times the Approval stage gets triggered simultaneously with the Source as soon as any code is pushed.
Weirdly enough this only happens to two pipelines that were created using a CloudFormation Template.
In a year or working with aws CodePipeline I never experienced such a thing.
Example of Approval stage getting skipped
Example of Approval stage getting triggered at the same time as the Source stage:
The CloudFormation Template:
AWSTemplateFormatVersion: 2010-09-09
Description: >-
Pipeline for React mobile WebApps. Creates a CodePipeline, along with a Deployment S3 that hosts the static and a CodeBuild Project to build and package the project
Parameters:
RepositoryName:
Type: String
Description: Name of repository to build from
RepositoryBranch:
Type: String
Description: The branch to pull from and build
Default: master
AllowedValues:
- master
- staging
ApiURL:
Type: String
Description: domain of the api to be used by the web app
SocketURL:
Type: String
Description: url of the socket to be used by the web app
SentryURL:
Type: String
Description: url of sentry
CloudfrontURL:
Type: String
Description: url of storage
Resources:
CodeBuildProject:
Type: AWS::CodeBuild::Project
DependsOn: CodeBuildRole
Properties:
Name: !Join
- '-'
- - !Ref RepositoryName
- !Ref RepositoryBranch
- build
- project
ServiceRole: !GetAtt CodeBuildRole.Arn
Environment:
Type: LINUX_CONTAINER
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/nodejs:8.11.0
EnvironmentVariables:
- Name: S3_URL
Value: !Join
- '-'
- - !Ref RepositoryName
- !Ref RepositoryBranch
- Name: ENV
Value: !Ref RepositoryBranch
- Name: API_DOMAIN
Value: !Ref ApiURL
- Name: SOCKET_URL
Value: !Ref SocketURL
- Name: SENTRY_URL
Value: !Ref SentryURL
- Name: AWS_CLOUDFRONT_URL
Value: !Ref CloudfrontURL
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.2
phases:
install:
commands:
- echo "installing dependencies"
- npm install
pre_build:
commands:
- echo "building static files"
- npm run build
- echo "static files finished building"
build:
commands:
- echo "build phase started"
- aws s3 sync ./dist s3://${DeploymentBucket}/ --cache-control max-age=0
- echo "build complete"
Artifacts:
Type: CODEPIPELINE
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: codebuild-service
PolicyDocument:
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
Version: '2012-10-17'
CodePipeline:
Type: 'AWS::CodePipeline::Pipeline'
DependsOn:
- CodePipelineTrustRole
Properties:
Stages:
- Name: Source
Actions:
- InputArtifacts: []
Name: Source
ActionTypeId:
Category: Source
Owner: AWS
Version: '1'
Provider: CodeCommit
OutputArtifacts:
- Name: MyApp
Configuration:
PollForSourceChanges: false
BranchName: !Ref RepositoryBranch
RepositoryName: !Ref RepositoryName
RunOrder: 1
- Name: Approval
Actions:
- InputArtifacts: []
Name: Approval
ActionTypeId:
Category: Approval
Owner: AWS
Version: '1'
Provider: Manual
OutputArtifacts: []
Configuration:
RunOrder: 1
- Name: BuildAndDeploy
Actions:
- InputArtifacts:
- Name: MyApp
Name: CodeBuild
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
OutputArtifacts:
- Name: MyAppBuild
Configuration:
ProjectName: !Ref CodeBuildProject
RunOrder: 1
- Name: Invalidation
Actions:
- InputArtifacts: []
Name: Invalidate-CloudFront
ActionTypeId:
Category: Invoke
Owner: AWS
Version: '1'
Provider: Lambda
Configuration:
FunctionName: "Cloudfront-Invalidator"
UserParameters: !Sub '{"S3Bucket": "${DeploymentBucket}"}'
OutputArtifacts: []
RunOrder: 1
ArtifactStore:
Type: S3
Location: pipeline-store-bucket
RoleArn: !GetAtt
- CodePipelineTrustRole
- Arn
Name: !Join
- '-'
- - !Ref RepositoryName
- !Ref RepositoryBranch
- pipeline
CodePipelineTrustRole:
Type: 'AWS::IAM::Role'
Description: Creates service role in IAM for AWS CodePipeline
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: 'sts:AssumeRole'
Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Sid: 1
Path: /
Policies:
- PolicyDocument:
Statement:
- Action:
- 's3:GetObject'
- 's3:GetObjectVersion'
- 's3:GetBucketVersioning'
- 's3:PutObject'
Effect: Allow
Resource:
- !Join
- ''
- - 'arn:aws:s3:::'
- 'pipeline-store-bucket'
- '/*'
- !Join
- ''
- - 'arn:aws:s3:::'
- 'pipeline-store-bucket'
- Action:
- 'codecommit:CancelUploadArchive'
- 'codecommit:GetBranch'
- 'codecommit:GetCommit'
- 'codecommit:GetUploadArchiveStatus'
- 'codecommit:UploadArchive'
Effect: Allow
Resource:
- !Join
- ':'
- - arn
- aws
- codecommit
- !Ref 'AWS::Region'
- !Ref 'AWS::AccountId'
- !Ref RepositoryName
- Action:
- 'codebuild:StartBuild'
- 'codebuild:BatchGetBuilds'
- 'codebuild:StopBuild'
Effect: Allow
Resource: '*'
- Action:
- 'cloudformation:DescribeStacks'
- 'cloudformation:DescribeChangeSet'
- 'cloudformation:CreateChangeSet'
- 'cloudformation:DeleteChangeSet'
- 'cloudformation:ExecuteChangeSet'
Effect: Allow
Resource: '*'
- Action:
- 'sns:Publish'
Effect: Allow
Resource: '*'
- Action:
- 'lambda:*'
- 'cloudwatch:*'
- 'events:*'
- 'codepipeline:PutJobSuccessResult'
- 'codepipeline:PutJobFailureResult'
Effect: Allow
Resource: '*'
PolicyName: CodePipelineTrustPolicy
RoleName: !Join
- '-'
- - !Ref AWS::StackName
- CodePipeline
- Role
SourceEvent:
Type: 'AWS::Events::Rule'
Properties:
Description: >-
Rule for Amazon CloudWatch Events to detect changes to the source
repository and trigger pipeline execution
EventPattern:
detail:
event:
- referenceCreated
- referenceUpdated
referenceName:
- !Ref RepositoryBranch
referenceType:
- branch
detail-type:
- CodeCommit Repository State Change
resources:
- !Join
- ':'
- - arn
- aws
- codecommit
- !Ref 'AWS::Region'
- !Ref 'AWS::AccountId'
- !Ref RepositoryName
source:
- aws.codecommit
Name: !Join
- '-'
- - !Ref RepositoryName
- !Ref RepositoryBranch
- SourceEvent
State: ENABLED
Targets:
- Arn: !Join
- ':'
- - arn
- aws
- codepipeline
- !Ref 'AWS::Region'
- !Ref 'AWS::AccountId'
- !Join
- '-'
- - !Ref RepositoryName
- !Ref RepositoryBranch
- pipeline
Id: ProjectPipelineTarget
RoleArn: !GetAtt SourceEventRole.Arn
SourceEventRole:
Type: 'AWS::IAM::Role'
Description: >-
IAM role to allow Amazon CloudWatch Events to trigger AWS CodePipeline
execution
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: 'sts:AssumeRole'
Effect: Allow
Principal:
Service:
- events.amazonaws.com
Sid: 1
Policies:
- PolicyDocument:
Statement:
- Action:
- 'codepipeline:StartPipelineExecution'
Effect: Allow
Resource:
- !Join
- ':'
- - arn
- aws
- codepipeline
- !Ref 'AWS::Region'
- !Ref 'AWS::AccountId'
- !Join
- '-'
- - !Ref RepositoryName
- !Ref RepositoryBranch
- pipeline
PolicyName: CodeStarWorkerCloudWatchEventPolicy
RoleName: !Join
- '-'
- - CodePipeline
- !Ref RepositoryName
- !Ref RepositoryBranch
- CloudWatchEventRule
DeploymentBucket:
Type: 'AWS::S3::Bucket'
Description: >-
S3 Bucket to host the website built from the CodeCommit repository
Properties:
AccessControl: PublicRead
WebsiteConfiguration:
IndexDocument: index.html
ErrorDocument: index.html
BucketName: !Join
- '-'
- - !Ref 'RepositoryName'
- !Ref 'RepositoryBranch'
DeletionPolicy: Delete
DeploymentBucketPolicy:
Type: AWS::S3::BucketPolicy
Description: >-
Policy for the web hosting deployment S3 bucket
Properties:
Bucket: !Ref DeploymentBucket
PolicyDocument:
Statement:
- Sid: PublicReadForGetBucketObjectsxw
Effect: Allow
Principal: '*'
Action: s3:GetObject
Resource: !Join ['', ['arn:aws:s3:::', !Ref 'DeploymentBucket', /*]]
CloudFrontDistribution:
Type: AWS::CloudFront::Distribution
Description: A CloudFront Distribution for the website hosting S3 buckets
DependsOn: DeploymentBucket
Properties:
DistributionConfig:
Origins:
- DomainName: !Join
- '.'
- - !Ref DeploymentBucket
- s3
- amazonaws
- com
Id: !Join
- '-'
- - S3
- !Ref DeploymentBucket
CustomOriginConfig:
HTTPPort: 80
HTTPSPort: 443
OriginProtocolPolicy: https-only
Enabled: true
DefaultCacheBehavior:
AllowedMethods:
- GET
- HEAD
- OPTIONS
- PUT
- POST
- PATCH
- DELETE
ForwardedValues:
QueryString: 'false'
Cookies:
Forward: none
TargetOriginId: !Join
- '-'
- - S3
- !Ref DeploymentBucket
ViewerProtocolPolicy: redirect-to-https
IPV6Enabled: true
DefaultRootObject: index.html
Outputs:
PipelineURL:
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${CodePipeline}
Description: URL for the CodePipeline of this stack
SiteUrl:
Value: !GetAtt [DeploymentBucket, WebsiteURL]
Description: URL for the S3 Website
CodeBuildUrl:
Value: !Sub https://eu-west-1.console.aws.amazon.com/codebuild/home?${AWS::Region}#/projects/${CodeBuildProject}/view
Description: URL for the CodeBuild Project of this stack
RepositoryUrl:
Value: !Sub https://eu-west-1.console.aws.amazon.com/codesuite/codecommit/repositories/${RepositoryName}/browse?region=eu-west-1
Description: URL for the repository of this stack

Answer 1

Each CodePipeline stage can have a different version in it so that you can eg. test a new version in beta in parallel with the previous version being deployed to prod. All actions within a stage run the same set of versions (artifacts) though.
CodePipeline will not de-duplicate based on source commit ID because often it's desirable to re-release the same source commit (eg. because the build will pull updated dependencies, or because you want to reset things to a clean state after eg. a rollback).
This means there can be different versions with the same source commit ID in different stages. You go to the "View history" page to verify this.
Example of Approval stage getting skipped
In this print screen I suspect there's 2 changes. C1 was previously approved and is now deployed in "BuildAndDeploy". C2 is the change showing Succeeded in "Source" and waiting for approval in "Approval".
Example of Approval stage getting triggered at the same time as the Source stage:
This is also a case of 2 version. There's one version in Source and another in Approval.