How do I get the source artifact from CodePipeline on CDK? - amazon-web-services

I'm using the official AWS documentation to create a pipeline using CDK: https://docs.aws.amazon.com/cdk/latest/guide/cdk_pipeline.html#cdk_pipeline_define (with a slight variation to the docs, where I used a CodeStar connection, as the code comments recommend)
This automatically creates a self-mutating pipeline, with three stages -- Source, Synth, and UpdatePipeline. That's great.
I would like to add a new stage with a CodeBuild action. I'd like the CodeBuild action to be based on the buildspec.yml file in the source directory.
On the console, I can easily do this by clicking "Add new stage", "Add action", and selecting the input artifact from the dropdown menu.
However, on CDK, with this recommended setup there's no easy way to get access to the input artifacts.
I managed to do it by forcing buildPipeline() and doing this:
import * as cdk from "#aws-cdk/core";
import {
CodePipeline,
ShellStep,
CodePipelineSource,
} from "#aws-cdk/pipelines";
import * as codebuild from "#aws-cdk/aws-codebuild";
import * as codepipelineActions from "#aws-cdk/aws-codepipeline-actions";
export class PipelineStack extends cdk.Stack {
public readonly source: cdk.CfnOutput
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const source = CodePipelineSource.connection("someuser/somerepo", "master", {
connectionArn: "arn:aws:codestar-connections:us-east-1:REDACTED:connection/REDACTED"
});
const synthShellStep = new ShellStep("Synth", {
input: source,
commands: [
"cd infrastructure",
"npm run ci",
"npm run build",
"npx cdk synth"
],
"primaryOutputDirectory": "infrastructure/cdk.out"
});
const pipeline = new CodePipeline(this, "Pipeline", {
pipelineName: "FancyPipeline",
synth: synthShellStep
});
// Need to build the pipeline to access the
// source artifact
pipeline.buildPipeline();
const sourceStage = pipeline.pipeline.stage("Source");
if (sourceStage) {
const sourceOutputs = sourceStage.actions[0].actionProperties.outputs;
if (sourceOutputs && sourceOutputs.length > 0) {
const sourceArtifact = sourceOutputs[0];
const codeBuildProject = new codebuild.PipelineProject(this, 'DockerBuildProject', {
environment: {
privileged: true
}
});
const buildAction = new codepipelineActions.CodeBuildAction({
actionName: 'DockerBuild',
project: codeBuildProject,
input: sourceArtifact,
environmentVariables: {
AWS_DEFAULT_REGION: {
value: this.region
},
AWS_ACCOUNT_ID: {
value: this.account
},
IMAGE_REPO_NAME: {
value: "somereponame"
},
IMAGE_TAG: {
value: "latest"
}
}
});
pipeline.pipeline.addStage({
stageName: "DockerBuildStage",
actions: [buildAction],
});
}
}
}
}
But this feels overall pretty awkward, and I can't call addStage() on the CodePipeline construct anymore. Surely there's a better way to do what I'm trying to do?
Any help/advice would be appreciated. Thanks.

The codepipelineActions.CodeBuildAction method accepts a parameter titled outputs which is for the list of output Artifacts for this action. TypeScript source here. I think it's easier to follow in the python version of the docs though (link here).

Related

Caching Lambda Docker images from a CDK stack in a pipeline

For simplicity, suppose we have a stack that contains a single lambda function created as a Docker image:
import { Stack, StackProps, Duration } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
export class FunStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const exampleFun = new lambda.DockerImageFunction(this, "ExampleFun", {
code: lambda.DockerImageCode.fromImageAsset("lambda/example_fun"),
timeout: Duration.seconds(10)
});
}
}
I'm omitting the contents of lambda/example_fun because it is straightforward, i.e., it contains a single .py file with some dummy handler and a Dockerfile that uses say public.ecr.aws/lambda/python:3.9 as base and uses the handler as cmd.
Now, especially if there were many such lambdas and/or they were large, a CDK pipeline such as the one you construct as part of the AWS CDK Workshop won't cache any of them. Concretely, let us have:
import * as cdk from 'aws-cdk-lib';
import * as codecommit from 'aws-cdk-lib/aws-codecommit';
import { Construct } from 'constructs';
import {CodeBuildStep, CodePipeline, CodePipelineSource} from "aws-cdk-lib/pipelines";
import { FunStack } from "./fun-stack";
import { Stage, StageProps } from "aws-cdk-lib";
export class FunPipelineStage extends Stage {
constructor(scope: Construct, id: string, props?: StageProps) {
super(scope, id, props);
new FunStack(this, 'Fun');
}
}
export class FunPipelineStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const repo = new codecommit.Repository(this, 'FunRepo', {
repositoryName: "FunRepo"
});
const pipeline = new CodePipeline(this, 'Pipeline', {
pipelineName: 'FunLambdaPipeline',
synth: new CodeBuildStep('SynthStep', {
input: CodePipelineSource.codeCommit(repo, 'master'),
installCommands: [
'npm install -g aws-cdk'
],
commands: [
'npm ci',
'npm run build',
'npx cdk synth'
]
})
});
const deploy = new FunPipelineStage(this, 'Deploy');
const deployStage = pipeline.addStage(deploy);
}
}
How should the pipeline be modified to allow us to cache the DockerImageFunction the pipeline generates when deploying?
If I'm reading correctly the documentation for Build caching in AWS CodeBuild and deducing correctly from the CDK docs for BuildSpec, I think I should use codebuild.BuildSpec.fromObject to specify the buildspec file.
With some experimentation, I'm able to do simple install and/or build commands via fromObject and the buildspec file, but can't quite figure out how to cache. In particular, how can the pipeline refer to the Docker image being built as part of the stack? The goal is that on each build, in case the Docker images haven't changed, they would be read from the cache and avoid being rebuilt.
Perhaps another alternative is to set up an ECR repository, somehow on each build check whether the hash of the built container is found and if not, build and push. However, I don't know how to concretely do this as I can't see how to refer to the Docker images built, if that makes sense.
Provide a partial buildspec with the partialBuildSpec prop and specify the caching method using the cache prop as shown in the module overview:
...
synth: new CodeBuildStep('SynthStep', {
input: CodePipelineSource.codeCommit(repo, 'master'),
installCommands: [
'npm install -g aws-cdk'
],
commands: [
'npm ci',
'npm run build',
'npx cdk synth'
],
partialBuildSpec: codebuild.BuildSpec.fromObject({
cache: {
paths: [ "path/to/cache/**/*" ]
}
}),
cache: codebuild.Cache.bucket(new s3.Bucket(this, 'Cache')),
})
Other than that, your premise is faulty: the containers will be built in any case - you can just make it faster. It's impossible to know whether the container hash changed without building it first. If you want to cache docker layers to make builds faster, you can try including /var/lib/docker/overlay2/**/* in the cache.
Possibly relevant:
https://github.com/aws/aws-cdk/issues/19157
https://github.com/aws/aws-cdk/issues/9080

AWS CDK and AppSync: Invalid principal in policy: "SERVICE":"appsync"

I'm trying to follow tutorials and use AWS's CDK CLI to deploy a stack using AppSync and some resource creations fail with errors like the following showing under events for the Stack in the CloudFormation console:
Invalid principal in policy: "SERVICE":"appsync" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 8d98f07c-d717-4dfe-af96-14f2d72d993f; Proxy: null)
I suspect what happened is that when cleaning up things on my personal developer account I deleted something I shouldn't have, but due to limited AWS experience I don't know what to create, I suspect it's an IAM policy, but I don't know the exact settings to use.
I'm trying on a new clean project created using cdk init sample-app --language=typescript. Running cdk deploy immediately after the above command works fine.
I originally tried using cdk-appsync-transformer to create GraphQL endpoints to a DynamoDB table and encountered the error.
I tried re-running cdk bootstrap after deleting the CDKToolkit CloudFormation stack, but it's not fixing this problem.
To rule out that it's due to something with the 3rd party library, I tried using AWS's own AppSync Construct Library instead and even following the example there I encountered the same error (although on creation of different resource types).
Reproduction steps
Create a new folder.
In the new folder run cdk init sample-app --language=typescript.
Install the AWS AppSync Construct Library: npm i #aws-cdk/aws-appsync-alpha#2.58.1-alpha.0 --save.
As per AWS's docs:
Create lib/schema.graphql with the following:
type demo {
id: String!
version: String!
}
type Query {
getDemos: [ demo! ]
}
input DemoInput {
version: String!
}
type Mutation {
addDemo(input: DemoInput!): demo
}
Update the lib/<projectName>-stack.ts file to be essentially like the following:
import * as appsync from '#aws-cdk/aws-appsync-alpha';
import { Duration, Stack, StackProps } from 'aws-cdk-lib';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as subs from 'aws-cdk-lib/aws-sns-subscriptions';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import { Construct } from 'constructs';
import * as path from 'path';
export class CdkTest3Stack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const queue = new sqs.Queue(this, 'CdkTest3Queue', {
visibilityTimeout: Duration.seconds(300)
});
const topic = new sns.Topic(this, 'CdkTest3Topic');
topic.addSubscription(new subs.SqsSubscription(queue));
const api = new appsync.GraphqlApi(this, 'Api', {
name: 'demo',
schema: appsync.SchemaFile.fromAsset(path.join(__dirname, 'schema.graphql')),
authorizationConfig: {
defaultAuthorization: {
authorizationType: appsync.AuthorizationType.IAM,
},
},
xrayEnabled: true,
});
const demoTable = new dynamodb.Table(this, 'DemoTable', {
partitionKey: {
name: 'id',
type: dynamodb.AttributeType.STRING,
},
});
const demoDS = api.addDynamoDbDataSource('demoDataSource', demoTable);
// Resolver for the Query "getDemos" that scans the DynamoDb table and returns the entire list.
// Resolver Mapping Template Reference:
// https://docs.aws.amazon.com/appsync/latest/devguide/resolver-mapping-template-reference-dynamodb. html
demoDS.createResolver('QueryGetDemosResolver', {
typeName: 'Query',
fieldName: 'getDemos',
requestMappingTemplate: appsync.MappingTemplate.dynamoDbScanTable(),
responseMappingTemplate: appsync.MappingTemplate.dynamoDbResultList(),
});
// Resolver for the Mutation "addDemo" that puts the item into the DynamoDb table.
demoDS.createResolver('MutationAddDemoResolver', {
typeName: 'Mutation',
fieldName: 'addDemo',
requestMappingTemplate: appsync.MappingTemplate.dynamoDbPutItem(
appsync.PrimaryKey.partition('id').auto(),
appsync.Values.projecting('input'),
),
responseMappingTemplate: appsync.MappingTemplate.dynamoDbResultItem(),
});
//To enable DynamoDB read consistency with the `MappingTemplate`:
demoDS.createResolver('QueryGetDemosConsistentResolver', {
typeName: 'Query',
fieldName: 'getDemosConsistent',
requestMappingTemplate: appsync.MappingTemplate.dynamoDbScanTable(true),
responseMappingTemplate: appsync.MappingTemplate.dynamoDbResultList(),
});
}
}
Run cdk deploy.

dependency cannot cross stage boundaries

I have an ApplicationStack which created a S3Bucket:
export class ApplicationStack extends Cdk.Stack {
public readonly websiteBucket : S3.Bucket;
constructor(scope: Construct, id: string, props: ApplicationStackProps) {
super(scope, id, props);
// Amazon S3 bucket to host the store website artifact
this.websiteBucket = new S3.Bucket(this, "eCommerceWebsite", {
bucketName: `${props.websiteDomain}-${account}-${region}`,
websiteIndexDocument: "index.html",
websiteErrorDocument: "error.html",
removalPolicy: Cdk.RemovalPolicy.DESTROY,
autoDeleteObjects: true,
accessControl: S3.BucketAccessControl.PRIVATE,
encryption: S3.BucketEncryption.S3_MANAGED,
publicReadAccess: false,
blockPublicAccess: S3.BlockPublicAccess.BLOCK_ALL,
});
// Create a dummy export.
// https://www.endoflineblog.com/cdk-tips-03-how-to-unblock-cross-stack-references
this.exportValue(this.websiteBucket.bucketArn);
...
...
...
}
}
I also defines ApplicationStage which contains above ApplicationStack
export class ApplicationStage extends Cdk.Stage {
public readonly websiteBucket : S3.Bucket;
constructor(scope: Construct, id: string, props: ApplicationStageProps) {
super(scope, id, props);
const applicationStack = new ApplicationStack(this, `eCommerceDatabaseStack-${props.stageName}`, {
stageName: props.stageName,
websiteDomain: props.websiteDomain,
});
this.websiteBucket = applicationStack.websiteBucket;
}
public getWebsiteBucket() {
return this.websiteBucket;
}
}
In my pipeline stack, I want to create application stage for each stage that need deploy the website artifact to its corresponding S3 bucket. This is a cross-account CI/CD pipeline, and I have 3 separate AWS accounts(Alpha, Gamma, Prod) for this website. Whenever I ship code out, the pipeline should deploy the new artifact to Alpha then Gamma then Prod, and the alpha.ecommerce.com, gamma.ecommerce.com, ecommerce.com should be updated in this order. The problem happens when reference the S3Bucket in S3DeployAction below:
export class CodePipelineStack extends CDK.Stack {
constructor(scope: CDK.App, id: string, props: CodePipelineStackProps) {
super(scope, id, props);
...
...
// Here the pipelineStageInfoList contains Gamma and Prod information.
pipelineStageInfoList.forEach((pipelineStage: PipelineStageInfo) => {
const applicationStage = new ApplicationStage(this, pipelineStage.stageName, {
stageName: pipelineStage.stageName,
pipelineName: props.pipelineName,
websiteDomain: props.websiteDomain,
env: {
account: pipelineStage.awsAccount,
region: pipelineStage.awsRegion,
},
});
const stage = pipeline.addStage(applicationStage);
// Here is what went wrong. It is trying to deploy the S3Bucket for that stage.
stage.addAction(
new codepipeline_actions.S3DeployAction({
actionName: "Deploy-Website",
input: outputWebsite,
bucket: applicationStage.getWebsiteBucket(),
})
);
});
}
...
...
...
}
Run cdk synthesize got below error:
/Users/yangliu/Projects/eCommerce/eCommerceWebsitePipelineCdk/node_modules/aws-cdk-lib/core/lib/deps.ts:39
throw new Error(`You cannot add a dependency from '${source.node.path}' (in ${describeStage(sourceStage)}) to '${target.node.path}' (in ${describeStage(targetStage)}): dependency cannot cross stage boundaries`);
^
Error: You cannot add a dependency from 'eCommerceWebsitePipelineCdk-CodePipeline-Stack' (in the App) to 'eCommerceWebsitePipelineCdk-CodePipeline-Stack/ALPHA/eCommerceDatabaseStack-ALPHA' (in Stage 'eCommerceWebsitePipelineCdk-CodePipeline-Stack/ALPHA'): dependency cannot cross stage boundaries
I think this means that I didn't pass the S3Bucket reference in the right way here.
How to fix it?
Update with my solution 2022-09-06
Based on matthew-bonig#'s advice, I am able to get this work.
I have a separate stack to be deployed to each account to create the S3 buckets and its required CloudFront distribution. Then my pipeline stack just focus on tracking my GitHub repository, and update S3 bucket whenever a new commit is pushed.
This usually occurs because your pipeline is running in a different account/region than your stacks created from the pipelineStageInfoList.
If they aren't in the same account/region then then simplest route is to manually set the s3 bucket names by a property on your 'InfoList' and forgo using references like you're trying to use. So you'd have to deploy everything first, then come back with an update afterwards that sets those values.
If they are, then you can try to set the pipeline stacks account/region directly like you are with the other stacks and that might help.

CDK Pipelines: Use Stack output in `post`step of stage

I have a pipeline stack with a stage. This stage contains multiple stacks. One of the stacks creates a Step Function. Now I would like to trigger that step function in the post of the stage (I created InvokeStepFunctionStep as a custom ICodePipelineActionFactory implementation for this).
This is from my pipeline stack code:
// TODO make this dynamic
const stepFunctionArn = "arn:aws:states:<FULL_ARN_OMITTED>";
pipeline.addStage(stage, {
post: [ new InvokeStepFunctionStep('step-function-invoke', {
stateMachine: sfn.StateMachine.fromStateMachineArn(this, 'StepFunctionfromArn',stepFunctionArn),
stateMachineInput: StateMachineInput.literal(stepFunctionsInput)
})]
});
Obviously the hard coded ARN is bad. I tried getting the ARN of the step function as a variable from the stage's stack. However this fails with
dependency cannot cross stage boundaries
I also tried using a CfnOutput for the ARN but when I try to use it via Fn.ImportValue the UpdatePipelineStep fails in CloudFormation with
No export named EdgePackagingStateMachineArn found
What is the recommended way to pass this information dynamically?
You could try using the CfnOutput.importValue property to reference CfnOutput value, which works for me. See below:
Service stack:
export class XxxStack extends Stack {
public readonly s3BucketName: CfnOutput;
constructor(scope: Construct, id: string, props?: StackProps) {
...
this.s3BucketName = new CfnOutput(
this,
's3BucketName',
{
exportName: `${this.stackName}-s3BucketName`,
value: s3Bucket.bucketName,
}
);
}
}
Stage class:
import { CfnOutput, Construct, Stage, StageProps } from '#aws-cdk/core';
export class CdkPipelineStage extends Stage {
public readonly s3BucketName: CfnOutput;
constructor(scope: Construct, id: string, props?: StageProps) {
super(scope, id, props);
const service = new XxxStack(
this,
'xxx',
{
...
}
);
this.s3BucketName = service.s3BucketName;
}
}
Pipeline stack:
import { CdkPipeline, SimpleSynthAction } from '#aws-cdk/pipelines';
const pipeline = new CdkPipeline(this, 'Pipeline', {...})
const preprod = new CdkPipelineStage(this, 'Staging', {
env: { account: PREPROD_ACCOUNT, region: PIPELINE_REGION },
});
// put validations for the stages
const preprodStage = pipeline.addApplicationStage(preprod);
preprodStage.addActions(
new ShellScriptAction({
actionName: 'TestService',
additionalArtifacts: [sourceArtifact],
rolePolicyStatements: [
new PolicyStatement({
effect: Effect.ALLOW,
actions: ['s3:getObject'],
resources: [
`arn:aws:s3:::${preprod.s3BucketName.importValue}/*`,
`arn:aws:s3:::${preprod.s3BucketName.importValue}`,
],
}),
],
useOutputs: {
ENV_S3_BUCKET_NAME: pipeline.stackOutput(
preprod.s3BucketName
),
},
...
}),
);
Note: my CDK version is
$ cdk --version
1.121.0 (build 026cb8f)
And I can confirm that CfnOutput.importValue also works for CDK version 1.139.0, and CDK version 2.8.0
Option 1: easy, not optimal.
Specify a name for your Step Function, and pass it to both the stack that creates it, and your invokation step. Build the ARN from the name.
This option isn't great because specifying physical names for CloudFormation resources has its disadvantages - mainly the inability to introduce any subsequent change that requires resource replacement, which may likely be necessary for a step function.
Option 2: more convoluted, but might be better long-term.
Create an SSM parameter with the step function's ARN from within the stack that creates the step function, then read the SSM parameter in your invokation step.
This will also require specifying a physical name for a resource - the SSM parameter, but you are not likely to require resource replacement for it, so it is less of an issue.

How to make a S3 deployment with the modern version of CodePipeline

I am trying to setup a brand new pipeline with the last version of AWS CDK for typescript (1.128).
The creation of the pipeline is pretty straight forward. I have added sources and build stages with no issues. The objective here is to have an automatic deployment of a static landing page.
So far I have this piece of code:
const landingPageStep = new ShellStep(`${PREFIX}LandingPageCodeBuildStep`, {
input: CodePipelineSource.connection(`${GIT_ORG}/vicinialandingpage`, GIT_MAIN, {
connectionArn: GIT_CONNECTION_ARN, // Created using the AWS console
}),
installCommands: [
'npm ci',
],
commands: [
'npm run build',
],
primaryOutputDirectory: 'out',
})
const pipeline = new CodePipeline(this, `${PREFIX}Pipeline`, {
pipelineName: `${PREFIX}Pipeline`,
synth: new ShellStep(`${PREFIX}Synth`, {
input: CodePipelineSource.connection(`${GIT_ORG}/viciniacdk`, GIT_MAIN, {
connectionArn: GIT_CONNECTION_ARN, // Created using the AWS console
}),
commands: [
'npm ci',
'npm run build',
'npx cdk synth',
],
additionalInputs: {
'landing_page': landingPageStep,
},
}),
});
The step I am not sure how to achieve it is how to deploy to S3 using the output of "landing_page". With previous versions of Pipelines there was a heavy use of Artifacts objects and CodePipelineActions, something similar to this where sourceOutput is an Artifact object:
const targetBucket = new s3.Bucket(this, 'MyBucket', {});
const pipeline = new codepipeline.Pipeline(this, 'MyPipeline');
const deployAction = new codepipeline_actions.S3DeployAction({
actionName: 'S3Deploy',
stage: deployStage,
bucket: targetBucket,
input: sourceOutput,
});
const deployStage = pipeline.addStage({
stageName: 'Deploy',
actions: [deployAction],
});
Now it is completely different since you have access to FileSet objects and apparently the build steps are intended to be used nesting outputs as the example above. Every output file is saved in a bucket with ugly file names, so it is not intended to be accessed directly neither.
I have seen some hacky approaches replacing ShellStep by CodeBuildStep and using as a postbuild command in the buildspec.yml file something like this:
aws s3 sync out s3://cicd-codebuild-static-website/
But it is resolved in the build stage and not in a deployment stage where it will be ideal to exist.
I have not seen anything insightful in the documentation so any suggestion is welcome. Thanks!
You can extend Step and implement ICodePipelineActionFactory. It's an interface that gets codepipeline.IStage and adds whatever actions you need to add.
Once you have the factory step, you pass it as either pre or post options of the addStage() method option.
Something close to the following should work:
class S3DeployStep extends Step implements ICodePipelineActionFactory {
constructor(private readonly provider: codepipeline_actions.JenkinsProvider, private readonly input: FileSet) {
}
public produceAction(stage: codepipeline.IStage, options: ProduceActionOptions): CodePipelineActionFactoryResult {
stage.addAction(new codepipeline_actions.S3DeployAction({
actionName: 'S3Deploy',
stage: deployStage,
bucket: targetBucket,
input: sourceOutput,
runOrder: options.runOrder,
}));
return { runOrdersConsumed: 1 };
}
}
// ...
pipeline.addStage(stage, {post: [new S3DeployStep()]});
But a way way way simpler method would be to use BucketDeployment to do it as part of the stack deployment. It creates a custom resource that copies data to a bucket from your assets or from another bucket. It won't get its own step in the pipeline and it will create a Lambda function under the hood, but it's simpler to use.