I have previously created a CodePipeline for building from a private GitHub Repo as below:
pipeline.addStage({
stageName: 'Source',
actions: [
new GitHubSourceAction({
actionName: 'Checkout',
owner: github.ardcOwner,
repo: github.raidoSrcRepo,
branch: "main",
oauthToken: SecretValue.secretsManager('personal-token' ),
output: outputSources,
trigger: GitHubTrigger.NONE,
}),
],
});
The repo in question is now public.
Given that no webhook is needed, since the trigger is set to NONE - how do I create new GitHubSourceAction without providing the oauthToken property?
The property is mandatory.
Is there a way to provide a "dummy" value to the GitHubSourceAction for oauthToken such that the account doesn't need to have an unnecessary/unused secret?
I tried to use a dummy value (as opposed to dummy "Secret"):
oauthToken: SecretValue.unsafePlainText("dummy"),
But that failed with "Could not access the GitHub repository".
It was then I remembered that CodePipeline doesn't do a clone, it downloads a zip archive of the HEAD of the repo.
Presumably, CodePipeline needs the secret to download the zip archive from GitHub.
So I don't think CodePipeline can be used with a public GitHub repo without using an OAuth token.
Related
Here's my problem. I'm currently struggling to run a basic shell script execution action in my pipeline.
The pipeline was created thru the Pipeline construct in #aws-cdk/aws-codepipeline
import { Artifact, IAction, Pipeline } from "#aws-cdk/aws-codepipeline"
const pipeline = new Pipeline(this, "backend-pipeline",{
...
});
Now, I'm running a cross deployment pipeline and would like to invoke a lambda right after it's been created. Before, a simple ShellScriptAction would've sufficed in the older (#aws-cdk/pipelines) package, but for some reason, both packages pipelines and aws-codepipeline are both maintained at the same time.
What I would like to know is how to run a simple basic command in the new (aws-codepipeline) package, ideally as an Action in a Stage.
Thanks in advance!
You would use a codebuild.PipelineProject in a codepipeline_actions.CodeBuildAction to run arbitrary shell commands in your pipeline. The CDK has several build tool constructs*, used in different places. pipelines.CodePipeline specializes in deploying CDK apps, while the lower-level codepipeline.Pipeline has a broader build capabilities:
Build tool construct
CloudFormation Resource
Can use where?
pipelines.ShellStep
AWS::CodeBuild::Project
pipelines.CodePipeline
pipelines.CodeBuildStep
AWS::CodeBuild::Project
pipelines.CodePipeline
codebuild.PipelineProject
AWS::CodeBuild::Project
codepipeline.Pipeline
codebuild.Project
AWS::CodeBuild::Project
codepipeline.Pipeline or standalone
In your case, the setup goes Pipeline > Stage > CodeBuildAction > PipelineProject.
// add to stage actions
new codepipeline_actions.CodeBuildAction({
actionName: 'CodeBuild',
project: new codebuild.PipelineProject(this, 'PipelineProject', {
buildSpec: codebuild.BuildSpec.fromObject({
version: '0.2',
phases: {
build: { commands: ['echo "[project foo] $PROJECT_FOO"'] },
},
}),
environmentVariables: {
PROJECT_FOO: {
value: 'Foo',
type: codebuild.BuildEnvironmentVariableType.PLAINTEXT,
},
},
}),
input: sourceOutput,
});
* The ShellScriptAction you mentioned is another one, now deprecated in v1 and removed from v2.
#aws-cdk/aws-codepipeline is for AWS Codepipeline. #aws-cdk/pipelines is for utilizing AWS Codepipeline to deploy CDK apps. Read more about the package and its justification here.
Regarding your question, you have some options here.
First of all, if you're looking for a simple CodeBuild action to run arbitrary commands, you can use CodeBuildAction.
There's a separate action specifically for invoking a lambda, too, it's LambdaInvokeAction.
Both are part of #aws-cdk/aws-codepipeline-actions module.
When creating a codebuild project with a codecommit source (as shown here: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-codebuild-readme.html#codecommitsource), how can I set the SourceVersion for the codecommit repository?
I can specify the source version manually in console, and cloudformation supports it (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-codebuild-project.html#cfn-codebuild-project-sourceversion), but appears to be no option to set this through CDK in the default codebuild project constructor.
You can set the SourceVersion with default node.childDefaultChild. That is a reference for CloudFormation template or CfnProject. For example to reference master-branch:
const projectCnf = project.node.defaultChild as CfnProject;
projectCnf.sourceVersion = 'refs/heads/master';
More details here: https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html
Try branchOrRef as documented here
const codeBuildProject = new codeBuild.Project(
this,
"appWebsiteCodeBuild",
{
projectName: "app-website-code-build",
source: codeBuild.Source.codeCommit({
repository: options.repository,
branchOrRef: "refs/heads/master",
}),
},
);
In AWS CodeBuild, it's incredibly easy to pass environment variables, as shown in the documentation. If I want to get the event trigger reference, I can use the variable CODEBUILD_WEBHOOK_TRIGGER, which is context-sensitive: for a GitHub PUSH event, this will be the branch name, but for a PULL_REQUEST_CREATED or PULL_REQUEST_UPDATED event, this will be the PR number.
So the problem is this: when using AWS CodePipeline, the CodeBuild project "source" is the CodePipeline instead of the GitHub webhook. Suddenly, CODEBUILD_WEBHOOK_TRIGGER is an empty string and doesn't seem to know about anything about the original GitHub webhook event that triggered the CodePipeline.
How does one access those environment variables using a CodeBuild project that is triggered by a CodePipeline? It seems to be a use case that AWS overlooked, so it might be a bug. Unfortunately, very difficult to submit a bug report with only a basic access account.
You are correct. In this particular case, CodePipeline is the one making start-build API call to start the build. CODEBUILD_WEBHOOK_TRIGGER is CodeBuild specific and will only be set when the webhook invokes CodeBuild.
If you want to know the webhook that triggered pipeline, you can use list-webhooks [1] API call with additional filters based on pipeline name to get the webhook details.
Ref:
[1] https://docs.aws.amazon.com/cli/latest/reference/codepipeline/list-webhooks.html
Edit 1:
I was wrong that list-webhooks will get you the required information. I did some tests and it only gives you the list of webhooks defined for the Source action.
The closest I can get is using "list-pipeline-executions" [2] CLI call in your CodeBuild buildspec.
If you run this command:
$ aws codepipeline list-pipeline-executions --pipeline-name <Pipeline_Name> --region us-east-1 --max-items 1
It will give you output similar to this:
{
"pipelineExecutionSummaries": [
{
"pipelineExecutionId": "ccdd87a0-41e4-4489-9332-0720dc526b37",
"status": "InProgress",
"startTime": 1573037463.245,
"lastUpdateTime": 1573037463.245,
"sourceRevisions": [
{
"actionName": "Source",
"revisionId": "4d3bcb17e4a71e3d4bf15215954172639716c326",
"revisionSummary": "Merge pull request #3 from shariqmus/readme-edits\n\nUpdate Code.py",
"revisionUrl": "https://github.com/shariqmus/hello-world/commit/4d3bcb17e4a71e3d4bf15215954172639716c326"
}
]
}
],
"NextToken": "eyJuZXh0VG9rZW4iOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAxfQ=="
}
The 'revisionSummary' has the PR details. You can filter this value using 'jq' [3], so the command in your build spec will look something like:
Make sure your CodeBuild project's service role has permission to do 'ListPipelineExecutions' on the Pipeline
Add the following in Buildspec 'Install' phase:
apt-get install jq
Add the following in Buildspec where you need to get the commit message:
COMMIT_MSG=$(aws codepipeline list-pipeline-executions --pipeline-name --max-items 1 | jq -r '.pipelineExecutionSummaries[0].sourceRevisions[0].revisionSummary')
echo $COMMIT_MSG
I hope this answer was helpful.
Ref:
[2] https://docs.aws.amazon.com/cli/latest/reference/codepipeline/list-pipeline-executions.html
[3] https://stedolan.github.io/jq/
I'm using AWS CodeBuild to build an application, it is configured to push the build artifacts to an AWS S3 bucket.
On inspecting the artifcats/objects in the S3 bucket I realised that the objects has been encrypted.
Is it possible to disable to encryption on the artifcats/objects?
There is now a checkbox named "Disable artifacts encryption" under the artifacts section which allows you to disable encryption when pushing artifacts to S3.
https://docs.aws.amazon.com/codebuild/latest/APIReference/API_ProjectArtifacts.html
I know this is an old post but I'd like to add my experience in this regard.
My requirement was to get front end assets from a code commit repository, build them and put them in s3 bucket. s3 bucket is further connected with cloudfront for serving the static front end content (written in react in my case).
I found that cloudfront is unable to serve KMS encrypted content as I found KMS.UnrecognizedClientException when I hit the cloudfront Url. I tried to fix that and disabling encryption on aws codebuild artifacts seemed to be the easiest solution when I found this
However, I wanted to manage this using aws-cdk. This code snippet in TypeScript may come handy if you're trying to solve the same issue using aws-cdk
Firstly, get your necessary imports. For this answer it'd be the following:
import * as codecommit from '#aws-cdk/aws-codecommit';
import * as codebuild from '#aws-cdk/aws-codebuild';
Then, I used the following snippet in a class that extends to cdk Stack
Note: The same should work if your class extends to a cdk Construct
// replace these according to your requirement
const frontEndRepo = codecommit.Repository
.fromRepositoryName(this, 'ImportedRepo', 'FrontEnd');
const frontendCodeBuild = new codebuild.Project(this, 'FrontEndCodeBuild', {
source: codebuild.Source.codeCommit({ repository: frontEndRepo }),
buildSpec: codebuild.BuildSpec.fromObject({
version: '0.2',
phases: {
build: {
commands: [
'npm install && npm run build',
],
},
},
artifacts: {
files: 'build/**/*'
}
}),
artifacts: codebuild.Artifacts.s3({
bucket: this.bucket, // replace with s3 bucket object
includeBuildId: false,
packageZip: false,
identifier: 'frontEndAssetArtifact',
name: 'artifacts',
encryption: false // added this to disable the encryption on codebuild
}),
});
Also to ensure that everytime I push a code in the repository, a build is triggered, I added the following snippet in the same class.
// add the following line in your imports if you're using this snippet
// import * as targets from '#aws-cdk/aws-events-targets';
frontEndRepo.onCommit('OnCommit', {
target: new targets.CodeBuildProject(frontendCodeBuild),
});
Note: This may not be a perfect solution, but it's working well for me till now. I'll update this answer if I find a better solution using aws-cdk
Artifact encryption cannot be disabled in AWS CodeBuild
Context:
I have a CodePipeline set up that uses CodeCommit and CodeBuild as its source and build phases.
My build includes a plugin (com.zoltu.git-versioning) that uses the Git commit history to dynamically create a build version number.
Issue:
This fails on the AWS pipeline because of it cannot find any Git information in the source used to perform the build.
Clearly the action used to checkout the source uses an export which omits the Git metadata and history.
Question:
How do I configure CodeCommit or CodePipeline to do a proper git clone? I've looked in the settings for both these components (as well as CodeBuild) and cannot find any configuration to set the command used by the checkout action.
Has anyone got CodePipeline builds working with a checkout containing full Git metadata?
This is currently not possible with the CodeCommit action in CodePipeline.
https://forums.aws.amazon.com/thread.jspa?threadID=248267
CodePipeline supports git full clone as of October:
https://aws.amazon.com/about-aws/whats-new/2020/09/aws-codepipeline-now-supports-git-clone-for-source-actions/
In your console, go to the source stage and edit.
You will have a new option to fully clone your git history.
full clone option
In Terraform you will have to add it to the source action's configuration:
configuration = {
RepositoryName = var.repository_name
BranchName = "master"
OutputArtifactFormat = "CODEBUILD_CLONE_REF"
}
More info:
https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-codecommit-gitclone.html
Yes, CodePipeline supports now a Git Full Clone.
You just need to do some extra steps: https://docs.aws.amazon.com/codepipeline/latest/userguide/troubleshooting.html#codebuild-role-connections
However, CodePipeline does not currently support dynamic branches, Pull Requests. See Dynamically change branches on AWS CodePipeline
Therefore, if you need to extend your pipeline for Pull Requests, I'd recommend the approach posted by Timothy Jones above.
There's one more related thing that's worth mentioning. CodeBuild has the Full Clone option as well.
As long as you do not use the Local Source cache option, the Git history is there.
When I tried to use the above mentioned cache option, I noticed that .git is not a directory. It's a file containing one line of text, e.g.:
gitdir: /codebuild/local-cache/workspace/9475b907226283405f08daf5401aba99ec6111f966ae2b921e23aa256f52f0aa/.git
I don't know why it's currently implemented like this but, it's confusing (at least for me) and I don't consider it to be the expected behavior.
Although CodePipeline doesn't natively support this, you can get the information by cloning the repository in CodeBuild.
To do this, you need to set the permissions correctly, then carefully clone the repository.
Permissions
To give the permissions to clone the repository you need to:
Give your CodeBuild role the codecommit:GitPull permission, with the resource ARN of your CodeCommit repository
Put git-credential-helper: yes in the env part of your buildspec file
Cloning the repo
To clone the repo, you'll need to:
know the clone URL and branch (CodeBuild doesn't know this information)
git reset back to the commit that CodeBuild is
building (otherwise you'll have a race condition between commits and builds).
git reset "$CODEBUILD_RESOLVED_SOURCE_VERSION"
If you'd like examples, I've made a detailed writeup of the process, and published an example CodePipeline stack showing it in action.
I spent too much time on this poorly documented process, that I decided to create some documentation for myself and future developers. I hope it helps.
CodeBuild + CodePipeline
This will connect CodeBuild and CodePipeline such that changes to your GitHub repository triggers CodePipeline to do a Full clone of your repository, that is then passed to CodeBuild which just transforms the local .git folder metadata to be poiting to the correct branch, and then all of the source code plus the Git metadata is deployed to Elastic Beanstalk.
More information about this process can be found here.
Start creating a CodePipeline pipeline. In the middle of its creation, you wull be prompted to create a CodeBuild project; do it.
Feel free to select a specific location for the Artifact store (custom S3 bucket).
Select GitHub (Version 2) as the source provider, check "Start the pipeline on source code change", and select Full cone as the output artifact format.
Select AWS CodeBuild as the Build provider.
For the Project Name, click onthe "Create project" button and select the below options:
a. Environment image: Managed image
b. Operating system: Amazon Linux 2
c. Runtime(s): Standard
d. For the Buildspec, select "Insert build commands" and click on "Switch to editor". Then paste the below Buildspec code.
e. Enable CloudWatch logs.
In the Environment variables, insert:
BranchName: #{SourceVariables.BranchName} as Plaintext
CommitId: #{SourceVariables.CommitId} as Plaintext
Select Single build as the Build type.
Select AWS Elastic Beanstalk as the Deploy provider.
Review operation and create the pipeline.
Create and add a new policy to the newly created CodeBuildServiceRole role. Choose a name, like projectName-connection-permission and attach the following JSON to it (tutorial):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "codestar-connections:UseConnection",
"Resource": "arn:aws:codestar-connections:eu-central-1:123456789123:connection/sample-1908-4932-9ecc-2ddacee15095"
}
]
}
PS: Change the Resource value arn:aws:codestar-connections:eu-central-1:123456789123:connection/sample-1908-4932-9ecc-2ddacee15095 from the JSON to your connection ARN. To find the connection ARN for your pipeline, open your pipeline and click the (i) icon on your source action.
Create and add a new policy to the newly created CodeBuildServiceRole role. Choose a name, like projectName-s3-access and attach the following JSON to it:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-s3-bucket-codepipeline",
"arn:aws:s3:::my-s3-bucket-codepipeline/*"
],
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetBucketAcl",
"s3:GetBucketLocation"
]
}
]
}
PS: Change the Resource values my-s3-bucket-codepipeline to match with your S3 bucket name for your CodePipeline.
Edit the inline policy for your CodePipelineServiceRole role by adding the following object to your Statement array:
{
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": "*"
}
Done.
Buildspec code
version: 0.2
#env:
#variables:
# key: "value"
# key: "value"
#parameter-store:
# key: "value"
# key: "value"
#secrets-manager:
# key: secret-id:json-key:version-stage:version-id
# key: secret-id:json-key:version-stage:version-id
#exported-variables:
# - variable
# - variable
#git-credential-helper: yes
#batch:
#fast-fail: true
#build-list:
#build-matrix:
#build-graph:
phases:
#install:
#If you use the Ubuntu standard image 2.0 or later, you must specify runtime-versions.
#If you specify runtime-versions and use an image other than Ubuntu standard image 2.0, the build fails.
#runtime-versions:
# name: version
# name: version
#commands:
# - command
# - command
#pre_build:
#commands:
# - command
# - command
build:
commands:
- echo Branch - $BranchName
- echo Commit - $CommitId
- echo Checking out branch - $BranchName
- git checkout $BranchName
# - command
# - command
#post_build:
#commands:
# - command
# - command
#reports:
#report-name-or-arn:
#files:
# - location
# - location
#base-directory: location
#discard-paths: yes
#file-format: JunitXml | CucumberJson
#artifacts:
#files:
# - location
# - location
#name: $(date +%Y-%m-%d)
#discard-paths: yes
#base-directory: location
artifacts:
files:
- '**/*'
#cache:
#paths:
# - paths
Additional Info
Never edit the inline policy that was created by CodePipeline! Only create and add new policies to a role. See this issue.
The Environment Variables for CodeBuild must be set from CodePipeline -> Edit: Build -> Environment variables - optional. If you set these variables in CodeBuild -> Edit -> Environment -> Additional configuration -> Environment variables it WON'T WORK!
For a bigger list of Environment variables during CodeBuild, see Variables List, Action Variables, and CodeBuild variables.
The Git Full clone option on CodePipeline is not available without CodeBuild. This is a known annoying limitation.
You can include the buildspec.yml in your root (top level) project directory. See this.
The Full clone that CodePipeline does leaves the local repository .git in a detached HEAD state, meaning that in order to get the branch name you will have to either get it with the help of CodeBuild environment variables to retrieve it from CodePipeline, or to execute the following command (see this):
git branch -a --contains HEAD | sed -n 2p | awk '{ printf $1 }'