Context:
I have a CodePipeline set up that uses CodeCommit and CodeBuild as its source and build phases.
My build includes a plugin (com.zoltu.git-versioning) that uses the Git commit history to dynamically create a build version number.
Issue:
This fails on the AWS pipeline because of it cannot find any Git information in the source used to perform the build.
Clearly the action used to checkout the source uses an export which omits the Git metadata and history.
Question:
How do I configure CodeCommit or CodePipeline to do a proper git clone? I've looked in the settings for both these components (as well as CodeBuild) and cannot find any configuration to set the command used by the checkout action.
Has anyone got CodePipeline builds working with a checkout containing full Git metadata?
This is currently not possible with the CodeCommit action in CodePipeline.
https://forums.aws.amazon.com/thread.jspa?threadID=248267
CodePipeline supports git full clone as of October:
https://aws.amazon.com/about-aws/whats-new/2020/09/aws-codepipeline-now-supports-git-clone-for-source-actions/
In your console, go to the source stage and edit.
You will have a new option to fully clone your git history.
full clone option
In Terraform you will have to add it to the source action's configuration:
configuration = {
RepositoryName = var.repository_name
BranchName = "master"
OutputArtifactFormat = "CODEBUILD_CLONE_REF"
}
More info:
https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-codecommit-gitclone.html
Yes, CodePipeline supports now a Git Full Clone.
You just need to do some extra steps: https://docs.aws.amazon.com/codepipeline/latest/userguide/troubleshooting.html#codebuild-role-connections
However, CodePipeline does not currently support dynamic branches, Pull Requests. See Dynamically change branches on AWS CodePipeline
Therefore, if you need to extend your pipeline for Pull Requests, I'd recommend the approach posted by Timothy Jones above.
There's one more related thing that's worth mentioning. CodeBuild has the Full Clone option as well.
As long as you do not use the Local Source cache option, the Git history is there.
When I tried to use the above mentioned cache option, I noticed that .git is not a directory. It's a file containing one line of text, e.g.:
gitdir: /codebuild/local-cache/workspace/9475b907226283405f08daf5401aba99ec6111f966ae2b921e23aa256f52f0aa/.git
I don't know why it's currently implemented like this but, it's confusing (at least for me) and I don't consider it to be the expected behavior.
Although CodePipeline doesn't natively support this, you can get the information by cloning the repository in CodeBuild.
To do this, you need to set the permissions correctly, then carefully clone the repository.
Permissions
To give the permissions to clone the repository you need to:
Give your CodeBuild role the codecommit:GitPull permission, with the resource ARN of your CodeCommit repository
Put git-credential-helper: yes in the env part of your buildspec file
Cloning the repo
To clone the repo, you'll need to:
know the clone URL and branch (CodeBuild doesn't know this information)
git reset back to the commit that CodeBuild is
building (otherwise you'll have a race condition between commits and builds).
git reset "$CODEBUILD_RESOLVED_SOURCE_VERSION"
If you'd like examples, I've made a detailed writeup of the process, and published an example CodePipeline stack showing it in action.
I spent too much time on this poorly documented process, that I decided to create some documentation for myself and future developers. I hope it helps.
CodeBuild + CodePipeline
This will connect CodeBuild and CodePipeline such that changes to your GitHub repository triggers CodePipeline to do a Full clone of your repository, that is then passed to CodeBuild which just transforms the local .git folder metadata to be poiting to the correct branch, and then all of the source code plus the Git metadata is deployed to Elastic Beanstalk.
More information about this process can be found here.
Start creating a CodePipeline pipeline. In the middle of its creation, you wull be prompted to create a CodeBuild project; do it.
Feel free to select a specific location for the Artifact store (custom S3 bucket).
Select GitHub (Version 2) as the source provider, check "Start the pipeline on source code change", and select Full cone as the output artifact format.
Select AWS CodeBuild as the Build provider.
For the Project Name, click onthe "Create project" button and select the below options:
a. Environment image: Managed image
b. Operating system: Amazon Linux 2
c. Runtime(s): Standard
d. For the Buildspec, select "Insert build commands" and click on "Switch to editor". Then paste the below Buildspec code.
e. Enable CloudWatch logs.
In the Environment variables, insert:
BranchName: #{SourceVariables.BranchName} as Plaintext
CommitId: #{SourceVariables.CommitId} as Plaintext
Select Single build as the Build type.
Select AWS Elastic Beanstalk as the Deploy provider.
Review operation and create the pipeline.
Create and add a new policy to the newly created CodeBuildServiceRole role. Choose a name, like projectName-connection-permission and attach the following JSON to it (tutorial):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "codestar-connections:UseConnection",
"Resource": "arn:aws:codestar-connections:eu-central-1:123456789123:connection/sample-1908-4932-9ecc-2ddacee15095"
}
]
}
PS: Change the Resource value arn:aws:codestar-connections:eu-central-1:123456789123:connection/sample-1908-4932-9ecc-2ddacee15095 from the JSON to your connection ARN. To find the connection ARN for your pipeline, open your pipeline and click the (i) icon on your source action.
Create and add a new policy to the newly created CodeBuildServiceRole role. Choose a name, like projectName-s3-access and attach the following JSON to it:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-s3-bucket-codepipeline",
"arn:aws:s3:::my-s3-bucket-codepipeline/*"
],
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetBucketAcl",
"s3:GetBucketLocation"
]
}
]
}
PS: Change the Resource values my-s3-bucket-codepipeline to match with your S3 bucket name for your CodePipeline.
Edit the inline policy for your CodePipelineServiceRole role by adding the following object to your Statement array:
{
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": "*"
}
Done.
Buildspec code
version: 0.2
#env:
#variables:
# key: "value"
# key: "value"
#parameter-store:
# key: "value"
# key: "value"
#secrets-manager:
# key: secret-id:json-key:version-stage:version-id
# key: secret-id:json-key:version-stage:version-id
#exported-variables:
# - variable
# - variable
#git-credential-helper: yes
#batch:
#fast-fail: true
#build-list:
#build-matrix:
#build-graph:
phases:
#install:
#If you use the Ubuntu standard image 2.0 or later, you must specify runtime-versions.
#If you specify runtime-versions and use an image other than Ubuntu standard image 2.0, the build fails.
#runtime-versions:
# name: version
# name: version
#commands:
# - command
# - command
#pre_build:
#commands:
# - command
# - command
build:
commands:
- echo Branch - $BranchName
- echo Commit - $CommitId
- echo Checking out branch - $BranchName
- git checkout $BranchName
# - command
# - command
#post_build:
#commands:
# - command
# - command
#reports:
#report-name-or-arn:
#files:
# - location
# - location
#base-directory: location
#discard-paths: yes
#file-format: JunitXml | CucumberJson
#artifacts:
#files:
# - location
# - location
#name: $(date +%Y-%m-%d)
#discard-paths: yes
#base-directory: location
artifacts:
files:
- '**/*'
#cache:
#paths:
# - paths
Additional Info
Never edit the inline policy that was created by CodePipeline! Only create and add new policies to a role. See this issue.
The Environment Variables for CodeBuild must be set from CodePipeline -> Edit: Build -> Environment variables - optional. If you set these variables in CodeBuild -> Edit -> Environment -> Additional configuration -> Environment variables it WON'T WORK!
For a bigger list of Environment variables during CodeBuild, see Variables List, Action Variables, and CodeBuild variables.
The Git Full clone option on CodePipeline is not available without CodeBuild. This is a known annoying limitation.
You can include the buildspec.yml in your root (top level) project directory. See this.
The Full clone that CodePipeline does leaves the local repository .git in a detached HEAD state, meaning that in order to get the branch name you will have to either get it with the help of CodeBuild environment variables to retrieve it from CodePipeline, or to execute the following command (see this):
git branch -a --contains HEAD | sed -n 2p | awk '{ printf $1 }'
Related
I am using AWS CDK Pipeline to deploy a simple 3-tier web application (defined with CDK as well).
The web app is inside a CodeCommit repository and I am referencing the repo in the cdk pipeline. So far so good...
Only particularity of the web app is that is composed of 3 folders and in one there is the cdk app, so when I am running the pipeline I need to cd into the folder before running the cdk commands. The structure cannot be changed. Below the code: (the 3-tier cdk app is written in Typescript, the pipeline in Python)
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
repo = codecommit.Repository.from_repository_name(self, "my-3tier-app",
repository_name="my-3tier-app"
)
pipeline = CodePipeline(self, "Pipeline_test",
pipeline_name="my-3tier-app-pipeline",
synth=ShellStep("Synth",
input=CodePipelineSource.code_commit(repo, "main"),
commands=[
"cd subfolder",
"npm install",
"npx cdk synth"
],
primary_output_directory="subfolder"
)
)
This creates a pipeline with 3 stages:
Source
Build
UpdatePipeline
The Build stage is where I can see the commands I defined in the Synth and it succeeds.
But I am having an hard time understanding the UpdatePipeline.
It fails with following error: Failed to store notices in the cache: Error: ENOENT: no such file or directory, open '/root/.cdk/cache/notices.json'.
In the stage details, build action, there is the following Buildsepc:
{
"version": "0.2",
"phases": {
"install": {
"commands": [
"npm install -g aws-cdk#2"
]
},
"build": {
"commands": [
"cdk -a . deploy PipelineStack --require-approval=never --verbose"
]
}
}
}
So I assume it is failing because it is running the cdk deploy from the wrong directory, not the subfolder.
If my assumption is correct, then I have the following questions:
Is this the default build action coming from the cdk?
How can I modify this Stage so I can add an extra cd subfolder to it? I am getting errors with an add_stage to the pipeline, but maybe they are not related...
or, how can I define the subfolder globally so I don't have to insert a lot of cds
Many thanks
You almost got it - you just need to adjust your primary_output_directory to point to the cdk.out folder: primary_output_directory="subfolder/cdk.out"
For reference, the CDK Pipelines docs mention this:
The pipeline assumes that your ShellStep will produce a cdk.out directory in the root, containing the CDK cloud assembly. If your CDK project lives in a subdirectory, be sure to adjust the primaryOutputDirectory to match
You already did this, you only need to point it to the cdk.out folder and it should work.
I've created a code pipeline for the PHP laravel base project with bitbucket. Passing parameter using AWS SSM to the appspec.yml All are working fine with the development branch. I need to update the parameters from the AWS SSM based on the branch name on appspec.yml file.
FOR DEV
Branch name: develop
parameter value: BRANCH_NAME_VALUE (develop_value)
FOR QA
Branch name: qa
parameter value: BRANCH_NAME_VALUE(qa_value)
appspec.yaml file
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html/
overwrite: true
hooks:
BeforeInstall:
- location: scripts/before_install.sh
timeout: 300
runas: root
AfterInstall:
- location: scripts/after_install.sh
timeout: 300
runas: root
How I can get the BRANCH_NAME for update the after_install.sh
Not sure what do you want to do, but you can't pass arbitrary env variables to CodeDeploy. The only supported ones are:
LIFECYCLE_EVENT : This variable contains the name of the lifecycle event associated with the script.
DEPLOYMENT_ID : This variables contains the deployment ID of the current deployment.
APPLICATION_NAME : This variable contains the name of the application being deployed. This is the name the user sets in the console or AWS CLI.
DEPLOYMENT_GROUP_NAME : This variable contains the name of the deployment group. A deployment group is a set of instances associated with an application that you target for a deployment.
DEPLOYMENT_GROUP_ID : This variable contains the ID of the deployment group in AWS CodeDeploy that corresponds to the current deployment
Thus, in your case you could have two development groups called develop and qa. Then, in the CodeDeploy scripts you could test the branch name using
DEPLOYMENT_GROUP_NAME and get respective SSM parameters.
Seems that you are trying to merge branches, and are facing issues where specific files or directories change per branch. I faced similar issue, and we can try to create .gitattributes per branch. The destination branch will have this so that once it is merged the specific files in the source branch wont overwrite the destination branch.
Check Reference:-
List item
https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes#_merge_strategies
List item
Git - Ignore files during merge
Example:-
2 branches master (For Production Environment)
and Stage (For Development Environment)
git config --global merge.ours.driver true
git checkout master
echo "appspec.yml merge=ours" >> .gitattributes
echo "scripts/before-install.sh merge=ours" >> .gitattributes
git merge stage
$ cat .gitattributes
appspec.yml merge=ours
scripts/before-install.sh merge=ours
Summary:-
so the idea is to keep the appspec.yml clean and environment free and handle it at git level itself. Unfortunately appspec.yml does not still support variables to accommodate per Branch.
Additionally, I would also add the above paths to .gitignore per branch to avoid them being altered during commits. Above is just a example, in production setup you could by default disable commits to master branch and only use pull requests with manual approval at AWS CodePipeline level with SNS topics for approval emails. And use a feature branch and merge to Stage first.
In AWS CodeBuild, it's incredibly easy to pass environment variables, as shown in the documentation. If I want to get the event trigger reference, I can use the variable CODEBUILD_WEBHOOK_TRIGGER, which is context-sensitive: for a GitHub PUSH event, this will be the branch name, but for a PULL_REQUEST_CREATED or PULL_REQUEST_UPDATED event, this will be the PR number.
So the problem is this: when using AWS CodePipeline, the CodeBuild project "source" is the CodePipeline instead of the GitHub webhook. Suddenly, CODEBUILD_WEBHOOK_TRIGGER is an empty string and doesn't seem to know about anything about the original GitHub webhook event that triggered the CodePipeline.
How does one access those environment variables using a CodeBuild project that is triggered by a CodePipeline? It seems to be a use case that AWS overlooked, so it might be a bug. Unfortunately, very difficult to submit a bug report with only a basic access account.
You are correct. In this particular case, CodePipeline is the one making start-build API call to start the build. CODEBUILD_WEBHOOK_TRIGGER is CodeBuild specific and will only be set when the webhook invokes CodeBuild.
If you want to know the webhook that triggered pipeline, you can use list-webhooks [1] API call with additional filters based on pipeline name to get the webhook details.
Ref:
[1] https://docs.aws.amazon.com/cli/latest/reference/codepipeline/list-webhooks.html
Edit 1:
I was wrong that list-webhooks will get you the required information. I did some tests and it only gives you the list of webhooks defined for the Source action.
The closest I can get is using "list-pipeline-executions" [2] CLI call in your CodeBuild buildspec.
If you run this command:
$ aws codepipeline list-pipeline-executions --pipeline-name <Pipeline_Name> --region us-east-1 --max-items 1
It will give you output similar to this:
{
"pipelineExecutionSummaries": [
{
"pipelineExecutionId": "ccdd87a0-41e4-4489-9332-0720dc526b37",
"status": "InProgress",
"startTime": 1573037463.245,
"lastUpdateTime": 1573037463.245,
"sourceRevisions": [
{
"actionName": "Source",
"revisionId": "4d3bcb17e4a71e3d4bf15215954172639716c326",
"revisionSummary": "Merge pull request #3 from shariqmus/readme-edits\n\nUpdate Code.py",
"revisionUrl": "https://github.com/shariqmus/hello-world/commit/4d3bcb17e4a71e3d4bf15215954172639716c326"
}
]
}
],
"NextToken": "eyJuZXh0VG9rZW4iOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAxfQ=="
}
The 'revisionSummary' has the PR details. You can filter this value using 'jq' [3], so the command in your build spec will look something like:
Make sure your CodeBuild project's service role has permission to do 'ListPipelineExecutions' on the Pipeline
Add the following in Buildspec 'Install' phase:
apt-get install jq
Add the following in Buildspec where you need to get the commit message:
COMMIT_MSG=$(aws codepipeline list-pipeline-executions --pipeline-name --max-items 1 | jq -r '.pipelineExecutionSummaries[0].sourceRevisions[0].revisionSummary')
echo $COMMIT_MSG
I hope this answer was helpful.
Ref:
[2] https://docs.aws.amazon.com/cli/latest/reference/codepipeline/list-pipeline-executions.html
[3] https://stedolan.github.io/jq/
I've been calling codebuild and manually overriding the buildspec like this:
aws codebuild start-build --cli-input-json file://servicea/custom.json
and then in custom.json
{
"projectName": "myproject",
"sourceVersion": "master",
"buildspecOverride": "servicea/buildspec.yml"
}
Now I want to use bitbucket trigger (or github if bitbucket is not supported) to build the service automatically after it's being pushed to master.
I've been Googling and found this tutorial https://docs.aws.amazon.com/codebuild/latest/userguide/sample-bitbucket-pull-request.html
However, I met a roadblock where I couldn't build a specific folder with a specific buildspec.
e.g.
for servicea, the build should run if I push to master and change any files in servicea folder with servicea/buildspec.yaml as the buildspec
for serviceb, the build should run if I push to master and change any files in serviceb folder with serviceb/buildspec.yaml as the buildspec
There is a FILE_PATH filter in the trigger, however there's I couldn't find a way to set the custom buildspec.
Is there any way to achieve this?
Note:
I want to use 1 codebuild project for all of my services
Bitbucket's webhook payload doesn't have the list of files changed in them, unlike GitHub.
Workaround:
Set the "git-credential-helper" to "yes" (or true) in your buildspec. Details in https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
You can then fetch the list of file changed for the specific commit using the call mentioned in https://community.atlassian.com/t5/Bitbucket-questions/Bitbucket-How-to-get-modified-files-of-a-commit-in-JSON-format/qaq-p/704126
You can obtain the commit from the environment variable: CODEBUILD_RESOLVED_SOURCE_VERSION and the branch from: CODEBUILD_WEBHOOK_HEAD_REF. Details in https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-env-vars.html
I'm using AWS CodeBuild to build an application, it is configured to push the build artifacts to an AWS S3 bucket.
On inspecting the artifcats/objects in the S3 bucket I realised that the objects has been encrypted.
Is it possible to disable to encryption on the artifcats/objects?
There is now a checkbox named "Disable artifacts encryption" under the artifacts section which allows you to disable encryption when pushing artifacts to S3.
https://docs.aws.amazon.com/codebuild/latest/APIReference/API_ProjectArtifacts.html
I know this is an old post but I'd like to add my experience in this regard.
My requirement was to get front end assets from a code commit repository, build them and put them in s3 bucket. s3 bucket is further connected with cloudfront for serving the static front end content (written in react in my case).
I found that cloudfront is unable to serve KMS encrypted content as I found KMS.UnrecognizedClientException when I hit the cloudfront Url. I tried to fix that and disabling encryption on aws codebuild artifacts seemed to be the easiest solution when I found this
However, I wanted to manage this using aws-cdk. This code snippet in TypeScript may come handy if you're trying to solve the same issue using aws-cdk
Firstly, get your necessary imports. For this answer it'd be the following:
import * as codecommit from '#aws-cdk/aws-codecommit';
import * as codebuild from '#aws-cdk/aws-codebuild';
Then, I used the following snippet in a class that extends to cdk Stack
Note: The same should work if your class extends to a cdk Construct
// replace these according to your requirement
const frontEndRepo = codecommit.Repository
.fromRepositoryName(this, 'ImportedRepo', 'FrontEnd');
const frontendCodeBuild = new codebuild.Project(this, 'FrontEndCodeBuild', {
source: codebuild.Source.codeCommit({ repository: frontEndRepo }),
buildSpec: codebuild.BuildSpec.fromObject({
version: '0.2',
phases: {
build: {
commands: [
'npm install && npm run build',
],
},
},
artifacts: {
files: 'build/**/*'
}
}),
artifacts: codebuild.Artifacts.s3({
bucket: this.bucket, // replace with s3 bucket object
includeBuildId: false,
packageZip: false,
identifier: 'frontEndAssetArtifact',
name: 'artifacts',
encryption: false // added this to disable the encryption on codebuild
}),
});
Also to ensure that everytime I push a code in the repository, a build is triggered, I added the following snippet in the same class.
// add the following line in your imports if you're using this snippet
// import * as targets from '#aws-cdk/aws-events-targets';
frontEndRepo.onCommit('OnCommit', {
target: new targets.CodeBuildProject(frontendCodeBuild),
});
Note: This may not be a perfect solution, but it's working well for me till now. I'll update this answer if I find a better solution using aws-cdk
Artifact encryption cannot be disabled in AWS CodeBuild