I am trying to use code pipeline in AWS CDK to automatically deploy code from GitHub source to S3 bucket. The code is as follows:
import * as codepipeline from '#aws-cdk/aws-codepipeline';
import * as codepipeline_actions from '#aws-cdk/aws-codepipeline-actions';
import * as s3 from '#aws-cdk/aws-s3';
import { Construct, Stack, StackProps } from '#aws-cdk/core';
export class S3PipelineStack extends Stack {
constructor(scope: Construct, id: string, props: StackProps = {}) {
super(scope, id, props);
const dagsBucket = s3.Bucket.fromBucketName(this, 'my-bucket', `test-bucket`);
const pipeline = new codepipeline.Pipeline(this, 'my-s3-pipeline', {
pipelineName: 'MyS3Pipeline',
});
const sourceOutput = new codepipeline.Artifact();
const sourceAction = new codepipeline_actions.CodeStarConnectionsSourceAction({
actionName: 'Source',
owner: '***',
repo: '***',
connectionArn: 'arn:aws:***',
output: sourceOutput,
branch: 'master',
});
const deployAction = new codepipeline_actions.S3DeployAction({
actionName: 'S3Deploy',
bucket: dagsBucket,
input: sourceOutput,
});
pipeline.addStage({
stageName: 'Source',
actions: [sourceAction],
});
pipeline.addStage({
stageName: 'Deploy',
actions: [deployAction],
});
}
}
This code works but the only problem is the S3 bucket can only add or change the code when source change in GitHub but cannot delete any code when anything deleted from source.
And I found a note in aws docs that said:
Another possible solution is from s3deploy.BucketDeployment but again it doesn't support connecting source from git but only can pass a source from local or another S3 bucket.
So does anybody know how to sync the GitHub and S3 bucket with add/change/delete from source in the right way?
Related
Here is my CodeBuild main page, which says "Artifacts upload location" is "alpha-artifact-bucket":
Here is one of the build run, which is not using above bucket:
What's the difference between the two? Why every build run use a random bucket?
Any way to enforce the CodeBuild use the specified S3 bucket "alpha-artifact-bucket"?
CDK code
CodeBuild stack: I deploy this stack to each AWS along the pipeline first, so that the pipeline stack will just query each AWS and find its corresponding CodeBuild, and add it as a "stage". The reason I'm doing this is because each AWS will have a dedicated CodeBuild stage which will need read some values from its SecretManger.
export interface CodeBuildStackProps extends Cdk.StackProps {
readonly pipelineName: string;
readonly pipelineRole: IAM.IRole;
readonly pipelineStageInfo: PipelineStageInfo;
}
/**
* This stack will create CodeBuild for the target AWS account.
*/
export class CodeBuildStack extends Cdk.Stack {
constructor(scope: Construct, id: string, props: CodeBuildStackProps) {
super(scope, id, props);
// DeploymentRole will be assumed by PipelineRole to perform the CodeBuild step.
const deploymentRoleArn: string = `arn:aws:iam::${props.env?.account}:role/${props.pipelineName}-DeploymentRole`;
const deploymentRole = IAM.Role.fromRoleArn(
this,
`CodeBuild${props.pipelineStageInfo.stageName}DeploymentRoleConstructID`,
deploymentRoleArn,
{
mutable: false,
// Causes CDK to update the resource policy where required, instead of the Role
addGrantsToResources: true,
}
);
const buildspecFile = FS.readFileSync("./config/buildspec.yml", "utf-8");
const buildspecFileYaml = YAML.parse(buildspecFile, {
prettyErrors: true,
});
new CodeBuild.Project(
this,
`${props.pipelineStageInfo.stageName}ColdBuild`,
{
projectName: `${props.pipelineStageInfo.stageName}ColdBuild`,
environment: {
buildImage: CodeBuild.LinuxBuildImage.STANDARD_5_0,
},
buildSpec: CodeBuild.BuildSpec.fromObjectToYaml(buildspecFileYaml),
role: deploymentRole,
logging: {
cloudWatch: {
logGroup: new Logs.LogGroup(
this,
`${props.pipelineStageInfo.stageName}ColdBuildLogGroup`,
{
retention: Logs.RetentionDays.ONE_WEEK,
}
),
},
},
}
);
}
}
Pipeline Stack:
export interface PipelineStackProps extends CDK.StackProps {
readonly description: string;
readonly pipelineName: string;
}
/**
* This stack will contain our pipeline..
*/
export class PipelineStack extends CDK.Stack {
private readonly pipelineRole: IAM.IRole;
constructor(scope: Construct, id: string, props: PipelineStackProps) {
super(scope, id, props);
// Get the pipeline role from pipeline AWS account.
// The pipeline role will assume "Deployment Role" of each AWS account to perform the actual deployment.
const pipelineRoleName: string =
"eCommerceWebsitePipelineCdk-Pipeline-PipelineRole";
this.pipelineRole = IAM.Role.fromRoleArn(
this,
pipelineRoleName,
`arn:aws:iam::${this.account}:role/${pipelineRoleName}`,
{
mutable: false,
// Causes CDK to update the resource policy where required, instead of the Role
addGrantsToResources: true,
}
);
// Initialize the pipeline.
const pipeline = new codepipeline.Pipeline(this, props.pipelineName, {
pipelineName: props.pipelineName,
role: this.pipelineRole,
restartExecutionOnUpdate: true,
});
// Add a pipeline Source stage to fetch source code from repository.
const sourceCode = new codepipeline.Artifact();
this.addSourceStage(pipeline, sourceCode);
// For each AWS account, add a build stage and a deployment stage.
pipelineStageInfoList.forEach((pipelineStageInfo: PipelineStageInfo) => {
const deploymentRoleArn: string = `arn:aws:iam::${pipelineStageInfo.awsAccount}:role/${props.pipelineName}-DeploymentRole`;
const deploymentRole: IAM.IRole = IAM.Role.fromRoleArn(
this,
`DeploymentRoleFor${pipelineStageInfo.stageName}`,
deploymentRoleArn
);
const websiteArtifact = new codepipeline.Artifact();
// Add build stage to build the website artifact for the target AWS.
// Some environment variables will be retrieved from target AWS's secret manager.
this.addBuildStage(
pipelineStageInfo,
pipeline,
deploymentRole,
sourceCode,
websiteArtifact
);
// Add deployment stage to for the target AWS to do the actual deployment.
this.addDeploymentStage(
props,
pipelineStageInfo,
pipeline,
deploymentRole,
websiteArtifact
);
});
}
// Add Source stage to fetch code from GitHub repository.
private addSourceStage(
pipeline: codepipeline.Pipeline,
sourceCode: codepipeline.Artifact
) {
pipeline.addStage({
stageName: "Source",
actions: [
new codepipeline_actions.GitHubSourceAction({
actionName: "Checkout",
owner: "yangliu",
repo: "eCommerceWebsite",
branch: "main",
oauthToken: CDK.SecretValue.secretsManager(
"eCommerceWebsite-GitHubToken"
),
output: sourceCode,
trigger: codepipeline_actions.GitHubTrigger.WEBHOOK,
}),
],
});
}
private addBuildStage(
pipelineStageInfo: PipelineStageInfo,
pipeline: codepipeline.Pipeline,
deploymentRole: IAM.IRole,
sourceCode: codepipeline.Artifact,
websiteArtifact: codepipeline.Artifact
) {
const stage = new CDK.Stage(this, `${pipelineStageInfo.stageName}BuildId`, {
env: {
account: pipelineStageInfo.awsAccount,
},
});
const buildStage = pipeline.addStage(stage);
const targetProject: CodeBuild.IProject = CodeBuild.Project.fromProjectName(
this,
`CodeBuildProject${pipelineStageInfo.stageName}`,
`${pipelineStageInfo.stageName}ColdBuild`
);
buildStage.addAction(
new codepipeline_actions.CodeBuildAction({
actionName: `BuildArtifactForAAAA${pipelineStageInfo.stageName}`,
project: targetProject,
input: sourceCode,
outputs: [websiteArtifact],
role: deploymentRole,
})
);
}
private addDeploymentStage(
props: PipelineStackProps,
pipelineStageInfo: PipelineStageInfo,
pipeline: codepipeline.Pipeline,
deploymentRole: IAM.IRole,
websiteArtifact: codepipeline.Artifact
) {
const websiteBucket = S3.Bucket.fromBucketName(
this,
`${pipelineStageInfo.websiteBucketName}ConstructId`,
`${pipelineStageInfo.websiteBucketName}`
);
const pipelineStage = new PipelineStage(this, pipelineStageInfo.stageName, {
stageName: pipelineStageInfo.stageName,
pipelineName: props.pipelineName,
websiteDomain: pipelineStageInfo.websiteDomain,
websiteBucket: websiteBucket,
env: {
account: pipelineStageInfo.awsAccount,
region: pipelineStageInfo.awsRegion,
},
});
const stage = pipeline.addStage(pipelineStage);
stage.addAction(
new codepipeline_actions.S3DeployAction({
actionName: `DeploymentFor${pipelineStageInfo.stageName}`,
input: websiteArtifact,
bucket: websiteBucket,
role: deploymentRole,
})
);
}
}
buildspec.yml:
version: 0.2
env:
secrets-manager:
REACT_APP_DOMAIN: "REACT_APP_DOMAIN"
REACT_APP_BACKEND_SERVICE_API: "REACT_APP_BACKEND_SERVICE_API"
REACT_APP_GOOGLE_MAP_API_KEY: "REACT_APP_GOOGLE_MAP_API_KEY"
phases:
install:
runtime-versions:
nodejs: 14
commands:
- echo Performing yarn install
- yarn install
build:
commands:
- yarn build
artifacts:
base-directory: ./build
files:
- "**/*"
cache:
paths:
- "./node_modules/**/*"
I figured this out. aws-codepipeline pipeline has a built-in artifacts bucket : CDK's CodePipeline or CodeBuildStep are leaving an S3 bucket behind, is there a way of automatically removing it?. That is different from the CodeBuild artifacts.
Because my pipeline role in Account A need to assume the deployment role in Account B to perform the CodeBuild step(of Account B), I need grant the deployment role in Account B the write permission to the pipeline's built-in artifacts bucket. So I need do this:
pipeline.artifactBucket.grantReadWrite(deploymentRole);
Basically, I want to build the pipeline that exists in one AWS acc, uses CodeCommit from another AWS acc, and deploys something in a third acc. I have this code for deploying my pipeline:
import * as codecommit from '#aws-cdk/aws-codecommit';
import * as codepipeline from '#aws-cdk/aws-codepipeline';
import * as codepipeline_actions from '#aws-cdk/aws-codepipeline-actions';
const repoArn = `arn:aws:codecommit:${vars.sourceRegion}:${vars.sourceAccount}:${vars.sourceRepo}`
const repo = codecommit.Repository.fromRepositoryArn(this, 'Source-repo', repoArn);
const sourceArtifact = new codepipeline.Artifact();
let trigger = CodeCommitTrigger.EVENTS
const sourceAction = new codepipeline_actions.CodeCommitSourceAction({
branch: vars.sourceBranch,
actionName: 'Source',
trigger: trigger,
output: sourceArtifact,
repository: repo,
variablesNamespace: 'SourceVariables',
codeBuildCloneOutput: true,
});
const pipelineBucket = s3.Bucket.fromBucketArn(this, 'pipelineBucket', BucketArn);
const pipeline = new codepipeline.Pipeline(this, 'CodePipeline', {
artifactBucket: pipelineBucket,
crossAccountKeys: true,
role: roles.codePiplineRole,
pipelineName: name,
stages: [
{
stageName: 'Source',
actions: [sourceAction],
},
],
});
If I run this I'll get the error: Source action 'Source' must be in the same region as the pipeline But they are both in the same region, and even in the same acc.
If I change codecommit.Repository.fromRepositoryArn to codecommit.Repository.fromRepositoryName then there will be no errors.
Is there any way to import an existing repo from ARN?
In the bin directory in my CDK project I have this:
#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from '#aws-cdk/core';
import {PipelineStack} from "../lib/pipeline-stack";
const app = new cdk.App();
new PipelineStack(app, 'PipelineStack', {
env: {account: '12345678912', region: 'us-east-1'},
});
app.synth();
Where PipelineStack is defined as (in my ../lib directory):
import {Construct, Stack, StackProps} from '#aws-cdk/core';
import {CodePipeline, CodePipelineSource, ShellStep} from '#aws-cdk/pipelines';
import {HadesStage} from './hades-stage';
/**
* The stack that defines the application pipeline
*/
export class PipelineStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const appName = 'MyApp';
const pipeline = new CodePipeline(this, 'Pipeline', {
// The pipeline name
pipelineName: 'MyAppPipeline',
// How it will be built and synthesized
synth: new ShellStep('Synth', {
// Where the source can be found
input: CodePipelineSource.gitHub('OWNER/REPO', 'master'),
// Install dependencies, build and run cdk synth
commands: [
'npm ci',
'npm run build',
'npx cdk synth'
],
}),
});
pipeline.addStage(new MyAppStage(this, 'MyAppProdStage', 'Prod', appName, 'mydomain.com', {
env: {account: '12345678912', region: 'us-east-1'}
}));
}
}
And MyAppStage is:
import {Construct, Stage, StageProps} from '#aws-cdk/core';
import {HadesStack} from './hades-stack';
/**
* Deployable unit of web service app
*/
export class MyAppStage extends Stage {
constructor(scope: Construct, id: string, stageName: string, appName: string, domainName: string, props?: StageProps) {
super(scope, id, props);
new MyAppStack(this, `${appName}${stageName}Stack`, stageName, appName, domainName, {
stackName: `${appName}${stageName}Stack`,
});
}
}
And MyAppStack is a stack with my actual resources. Basically I followed this guide.
It works fine, until I add a secret rotation for RDS credentials. The MyAppStack stack fails with:
Requires capabilities : [CAPABILITY_AUTO_EXPAND]
Which makes sense; however, I can't find a way to add the IAM capabilities to the stack through CDK. Am I doing something wrong? Or is the approach covered by the guide not meant to cover this? Can I add the capabilities somehow?
It appears that this was, indeed, a bug, and Otavio Macedo fixed it in GitHub pull request #15819.
When I create a codebuild project in AWS console, I can select AWS CodePipeline as source provider. See below screenshot.
But in CDK, https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-codebuild.Source.html, I can't find how I can specify AWS CodePipeline as source provider. How can I achieve it in CDK?
Seems like you assign a codebuild action to the codepipeline instead of adding a codepipeline to the codebuild project source.
https://docs.aws.amazon.com/cdk/api/latest/docs/aws-codebuild-readme.html#codepipeline
To add a CodeBuild Project as an Action to CodePipeline, use the PipelineProject class instead of Project. It's a simple class that doesn't allow you to specify sources, secondarySources, artifacts or secondaryArtifacts, as these are handled by setting input and output CodePipeline Artifact instances on the Action, instead of setting them on the Project.
https://docs.aws.amazon.com/cdk/api/latest/docs/aws-codepipeline-actions-readme.html#build--test
Example of a CodeBuild Project used in a Pipeline, alongside CodeCommit:
import * as codebuild from '#aws-cdk/aws-codebuild';
import * as codecommit from '#aws-cdk/aws-codecommit';
import * as codepipeline_actions from '#aws-cdk/aws-codepipeline-actions';
const repository = new codecommit.Repository(this, 'MyRepository', {
repositoryName: 'MyRepository',
});
const project = new codebuild.PipelineProject(this, 'MyProject');
const sourceOutput = new codepipeline.Artifact();
const sourceAction = new codepipeline_actions.CodeCommitSourceAction({
actionName: 'CodeCommit',
repository,
output: sourceOutput,
});
const buildAction = new codepipeline_actions.CodeBuildAction({
actionName: 'CodeBuild',
project,
input: sourceOutput,
outputs: [new codepipeline.Artifact()], // optional
executeBatchBuild: true // optional, defaults to false
});
new codepipeline.Pipeline(this, 'MyPipeline', {
stages: [
{
stageName: 'Source',
actions: [sourceAction],
},
{
stageName: 'Build',
actions: [buildAction],
},
],
});
I wanna translate this CloudFormation piece into CDK:
Type: AWS::S3::BucketPolicy
Properties:
Bucket:
Ref: S3BucketImageUploadBuffer
PolicyDocument:
Version: "2012-10-17"
Statement:
Action:
- s3:PutObject
- s3:PutObjectAcl
Effect: Allow
Resource:
- ...
Looking at the documentation here, I don't see a way to provide the policy document itself.
This is an example from a working CDK-Stack:
artifactBucket.addToResourcePolicy(
new PolicyStatement({
resources: [
this.pipeline.artifactBucket.arnForObjects("*"),
this.pipeline.artifactBucket.bucketArn],
],
actions: ["s3:List*", "s3:Get*"],
principals: [new ArnPrincipal(this.deploymentRole.roleArn)]
})
);
Building on #Thomas Wagner's answer, this is how I did this. I was trying to limit the bucket to a given IP range:
import * as cdk from '#aws-cdk/core';
import * as s3 from '#aws-cdk/aws-s3';
import * as s3Deployment from '#aws-cdk/aws-s3-deployment';
import * as iam from '#aws-cdk/aws-iam';
export class StaticSiteStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// Bucket where frontend site goes.
const mySiteBucket = new s3.Bucket(this, 'mySiteBucket', {
websiteIndexDocument: "index.html"
});
let ipLimitPolicy = new iam.PolicyStatement({
actions: ['s3:Get*', 's3:List*'],
resources: [mySiteBucket.arnForObjects('*')],
principals: [new iam.AnyPrincipal()]
});
ipLimitPolicy.addCondition('IpAddress', {
"aws:SourceIp": ['1.2.3.4/22']
});
// Allow connections from my CIDR
mySiteBucket.addToResourcePolicy(ipLimitPolicy);
// Deploy assets
const mySiteDeploy = new s3Deployment.BucketDeployment(this, 'deployAdminSite', {
sources: [s3Deployment.Source.asset("./mysite")],
destinationBucket: mySiteBucket
});
}
}
I was able to use the s3.arnForObjects() and iam.AnyPrincipal() helper functions rather than specifying ARNs or Principals directly.
The assets I want to deploy to the bucket are kept in the root of my project directory in a directory called mysite, and then referenced via a call to s3Deployment.BucketDeployment. This can be any directory your build process has access to, of course.
The CDK does this a little differently. I believe you are supposed to use bucket.addToResourcePolicy, as documented here.
As per the original question, then the answer from #thomas-wagner is the way to go.
If anyone comes here looking for how to create the bucket policy for a CloudFront Distribution without creating a dependency on a bucket then you need to use the L1 construct CfnBucketPolicy (rough C# example below):
IOriginAccessIdentity originAccessIdentity = new OriginAccessIdentity(this, "origin-access-identity", new OriginAccessIdentityProps
{
Comment = "Origin Access Identity",
});
PolicyStatement bucketAccessPolicy = new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.ALLOW,
Principals = new[]
{
originAccessIdentity.GrantPrincipal
},
Actions = new[]
{
"s3:GetObject",
},
Resources = new[]
{
Props.OriginBucket.ArnForObjects("*"),
}
});
_ = new CfnBucketPolicy(this, $"bucket-policy", new CfnBucketPolicyProps
{
Bucket = Props.OriginBucket.BucketName,
PolicyDocument = new PolicyDocument(new PolicyDocumentProps
{
Statements = new[]
{
bucketAccessPolicy,
},
}),
});
Where Props.OriginBucket is an instance of IBucket (just a bucket).