CodeBuild failed for cross-account CI/CD pipeline - amazon-web-services

Question

I'm trying to follow this guide to setup a cross accounts pipeline: https://taimos.de/blog/create-a-cicd-pipeline-for-your-cdk-app
I have Alpha account 111111 which will hold the pipeline, that is going to deploy to Alpha account 111111 and Prod account 222222. I did the bootstrap below to establish a trust relationship to the CI/CD account to allow cross-account deployments.
cdk bootstrap --profile=eCommerceService-Prod --bootstrap-customer-key --cloudformation-execution-policies 'arn:aws:iam::aws:policy/AdministratorAccess' --trust 111111 --trust-for-lookup 111111 aws://222222/us-east-1
The pipeline is setup correctly, however, the ColdBuild failed:
User: arn:aws:sts::111111:assumed-role/eCommerceDatabaseCdk-Pipe-PipelineBuildSynthCdkBui-JE7GJT5LX3SC/AWSCodeBuild-32197845-1f33-44c4-a9ff-d33aae93448e is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222:role/cdk-hnb659fds-lookup-role-222222-us-east-1 . Please make sure that this role exists in the account. If it doesn't exist, (re)-bootstrap the environment with the right '--trust', using the latest version of the CDK CLI.
I checked that arn:aws:iam::222222:role/cdk-hnb659fds-lookup-role-222222-us-east-1 exists in prod account. So it seems to me that there is a missing permission in the codebuild step. However, I did not find anything particular for CodeBuild step in this blog: https://taimos.de/blog/create-a-cicd-pipeline-for-your-cdk-app.
The permissions for role arn:aws:iam::222222:role/cdk-hnb659fds-lookup-role-222222-us-east-1 mentioned in the error message is here:
It's trust entities is:
{
"Version": "2008-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111:root"
},
"Action": "sts:AssumeRole"
}
]
}
Any idea what am I missing here?
Code
Pipeline config:
export const pipelineStageInfoList: PipelineStageInfo[] = [
{
stageName: "ALPHA",
awsAccount: "111111",
awsRegion: "us-east-1"
},
{
stageName: "PROD",
awsAccount: "222222",
awsRegion: "us-east-1"
}
]
Pipeline stack:
export class PipelineStack extends Cdk.Stack {
constructor(scope: Cdk.App, id: string, props: PipelineStackProps) {
super(scope, id, props);
// Initialize the pipeline
const pipeline = new pipelines.CodePipeline(this, "Pipeline", {
pipelineName: "eCommerceDatabasePipeline",
// Create KMS keys for the artifact buckets,
// allowing cross-account deployments
crossAccountKeys: true,
// allow the pipeline to reconfigure itself when assets or stages
// are being added to it
selfMutation: true,
// synth is expected to produce the CDK Cloud Assembly as its output
synth: new pipelines.ShellStep("Synth", {
input: pipelines.CodePipelineSource.gitHub(
"pandanyc/eCommerceDatabaseCdk",
"main",
{
authentication: Cdk.SecretValue.secretsManager('github-token')
}
),
// Install dependencies, build and run cdk synth
commands: [
'npm ci',
'npm run build',
'npx cdk synth'
],
}),
});
// Add stages to this pipeline.
pipelineStageInfoList.forEach((pipelineStage: PipelineStageInfo) => {
pipeline.addStage(
new ApplicationStage(this, pipelineStage.stageName, {
stageName: pipelineStage.stageName,
pipelineName: props.pipelineName,
env: {
account: pipelineStage.awsAccount,
region: pipelineStage.awsRegion,
},
})
);
});
}
}

Answer 1

If you want to perform lookups in the pipeline itself, your synth step has to have the explicit permission to assume the lookup role. You would add it like this:
synth: new pipelines.CodeBuildStep("Synth", {
input: pipelines.CodePipelineSource.gitHub(
"pandanyc/eCommerceDatabaseCdk",
"main",
{
authentication: Cdk.SecretValue.secretsManager('github-token')
}
),
// Install dependencies, build and run cdk synth
commands: [
'npm ci',
'npm run build',
'npx cdk synth'
],
rolePolicyStatements: [ iam.PolicyStatement({
effect: iam.Effect.ALLOW,
actions: ['sts:AssumeRole'],
resources: ['*'],
conditions: { StringEquals: { 'iam:ResourceTag/aws-cdk:bootstrap-role': 'lookup' } }
}]
}),
It is also worth noting, however, that this is advised against in the Best Practices. Here's an excerpt, but I suggest you check out the whole thing:
AWS CDK includes a mechanism called context providers to record a
snapshot of non-deterministic values, allowing future synthesis
operations produce exactly the same template. The only changes in the
new template are the changes you made in your code. When you use a
construct's .fromLookup() method, the result of the call is cached in
cdk.context.json, which you should commit to version control along
with the rest of your code to ensure future executions of your CDK app
use the same value.
What this means in practice is that you should run cdk synth once locally, which will perform all the necessary lookups and store the results in cdk.context.json. You should then commit that, and your pipeline will use these cached values instead of doing the lookup every time.
As a result, your synth step also won't need to assume the lookup role.