How to run Parallel builds with AWS Codebuild? - amazon-web-services

Question

I am trying to setup a codebuild where I have one module that would create the dependencies for other builds. And these other builds will use the artifacts generated by the first build and these build should be ran in parallel. I was looking at documentation of CodeBuild it seems there isn't information there. Sample of my buildspec.yml
version: 0.2
#env:
#variables:
# key: "value"
# key: "value"
#parameter-store:
# key: "value"
# key: "value"
#git-credential-helper: yes
phases:
#install:
#If you use the Ubuntu standard image 2.0 or later, you must specify runtime-versions.
#If you specify runtime-versions and use an image other than Ubuntu standard image 2.0, the build fails.
#runtime-versions:
# name: version
# name: version
#commands:
# - command
# - command
# pre_build:
# commands:
# - mkdir artifacts
# - pwd
# - ls
build:
commands:
- cd frontend
- npm install
- cd ..
- cd othermodule
- npm install
#post_build:
#commands:
# - cp -a $CODEBUILD_SRC_DIR/frontend/dist.
# - command
artifacts:
files:
- package-lock.json
- node_modules/*
base-directory: frontend
#cache:
#paths:
# - paths

Answer 1

CodeBuild is used to automate build step, not to automate the whole CICD process. In CodeBuild, you specify buildspec.yml to automate a sequence of steps that need to be performed in that particular build.
If you need to automate sequence of builds then the easiest option that you have is to use CodePipeline where you can create stage for each step in your CICD process. In your case, this would mean that one step (stage) would be CodeBuild action that you have described in your post which would transition into another stage where you can specify another CodeBuild actions and these actions can be specified to take artifacts from the previous step as an input and you can run them in parallel.
So it would look like this
INPUT -> STAGE (perform initial build) -> STAGE (specify multiple build actions in parallel - in console, this can be done by placing them next to each other horizontally instead of vertically)
The other option, without using CodePipeline, would be to use Lambda function with CloudWatch events. CodeBuild publishes event once the build is done. You can subscribe Lambda function to that event and write a code that would execute following builds as needed.

Answer 2

As #MEMark has pointed out, currently AWS CodePipeline service supports Bitbuket.
One suitable approach to reach Bitbucket from AWS is by providing an Star Connection(Developer Tools > Code Build > Settings > Connections).
Once the connection is created the pipeline can be deployed using for instance following code:
AWSTemplateFormatVersion: "2010-09-09"
Description: TBD
Parameters:
DeployStage:
Type: String
Description: name of the stage to deploy(dev, test, prod)
BranchName:
Type: String
Description: TBD
CodeStarConnectionARN:
Type: String
Description: ARN of the codestar connection to Bitbucket.
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
Effect: Allow
Principal:
Service: codebuild.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
PipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: 'sts:AssumeRole'
Effect: Allow
Principal:
Service: codepipeline.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyName: CodePipelineAccess
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- TBD
- ...
Effect: Allow
Resource: '*'
-
Effect: Allow
Action:
- TBD
- ...
Resource: "*"
BuildUi:
Type: AWS::CodeBuild::Project
Properties:
Name: !Join ['', ['build-ui-', !Ref DeployStage]]
Description: TBD
ServiceRole: !Ref CodeBuildRole
Artifacts:
Type: CODEPIPELINE
Environment:
Type: LINUX_CONTAINER
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/standard:5.0
EnvironmentVariables:
- Name: DEPLOY_STAGE
Type: PLAINTEXT
Value: !Ref DeployStage
- Name: DEPLOY_BRANCH_NAME
Type: PLAINTEXT
Value: !Ref BranchName
Source:
Type: CODEPIPELINE
Location: https://user#bitbucket.org/organization/projectName.git
BuildSpec: buildspec.ui.yml
SourceVersion: !Ref BranchName
Tags:
-
Key: cost
Value: ui_build
TimeoutInMinutes: 10
BuildApi:
Type: AWS::CodeBuild::Project
Properties:
Name: !Join ['', ['build-api-', !Ref DeployStage]]
Description: TBD
ServiceRole: !Ref CodeBuildRole
Artifacts:
Type: CODEPIPELINE
Environment:
Type: LINUX_CONTAINER
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/standard:5.0
EnvironmentVariables:
- Name: DEPLOY_STAGE
Type: PLAINTEXT
Value: !Ref DeployStage
- Name: DEPLOY_BRANCH_NAME
Type: PLAINTEXT
Value: !Ref BranchName
Source:
Type: CODEPIPELINE
Location: https://user#bitbucket.org/organization/projectName.git
BuildSpec:buildspec.api.yml
SourceVersion: !Ref BranchName
Tags:
-
Key: cost
Value: api_build
TimeoutInMinutes: 10
ArtifactStoreBucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: 'BucketOwnerFullControl'
VersioningConfiguration:
Status: Enabled
NamePipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
ArtifactStore:
Location: !Ref ArtifactStoreBucket
Type: S3
Name: !Join ['', ['pipeline-', !Ref DeployStage]]
RoleArn: !GetAtt PipelineRole.Arn
Stages:
- Name: Source
Actions:
- Name: Source
ActionTypeId:
Category: Source
Owner: AWS
Provider: CodeStarSourceConnection
Version: '1'
Configuration:
ConnectionArn: !Ref CodeStarConnectionARN
FullRepositoryId: "organization/projectName"
BranchName: !Ref BranchName
OutputArtifacts:
- Name: project-source
RunOrder: '1'
- Name: ParallelBuild
Actions:
- Name: BuildUi
ActionTypeId:
Category: Build
Owner: AWS
Version: 1
Provider: CodeBuild
OutputArtifacts:
- Name: project-build-ui
InputArtifacts:
- Name: project-source
Configuration:
ProjectName: !Ref BuildUi
RunOrder: 1
- Name: BuildApi
ActionTypeId:
Category: Build
Owner: AWS
Version: 1
Provider: CodeBuild
OutputArtifacts:
- Name: build-api
InputArtifacts:
- Name: project-source
Configuration:
ProjectName: !Ref BuildApi
RunOrder: 1
Notes:
it assumes the repository has this git url: https://user#bitbucket.org/organization/projectName.git
CodeBuildRole and PipelineRole should be defined properly for the use case