Fargate Service Cannot Write to DynamoDB Table (CDK) - amazon-web-services

I'm working through setting up a new infrastructure with the AWS CDK and I'm trying to get a TypeScript app running in Fargate to be able to read/write from/to a DynamoDB table, but am hitting IAM issues.
I have both my Fargate service and my DynamoDB Table defined, and both are running as they should be in AWS, but whenever I attempt to write to the table from my app, I am getting an access denied error.
I've tried the solutions defined in this post, as well as the ones it links to, but nothing seems to be allowing my container to write to the table. I've tried everything from setting table.grantReadWriteData(fargateService.taskDefinition.taskRole) to the more complex solutions described in the linked articles of defining my own IAM policies and setting the effects and actions, but I always just get the same access denied error when attempting to do a putItem:
AccessDeniedException: User: {fargate-service-arn} is not authorized to perform: dynamodb:PutItem on resource: {dynamodb-table} because no identity-based policy allows the dynamodb:PutItem action
Am I missing something, or a crucial step to make this possible?
Any help is greatly appreciated.
Thanks!
Edit (2022-09-19):
Here is the boiled down code for how I'm defining my Vpc, Cluster, Container Image, FargateService, and Table.
export class FooCdkStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const vpc = new Vpc(this, 'FooVpc', {
maxAzs: 2,
natGateways: 1
});
const cluster = new Cluster(this, 'FooCluster', { vpc });
const containerImage = ContainerImage.fromAsset(
path.join(__dirname, '/../app'),
{
platform: Platform.LINUX_AMD64 // I'm on an M1 Mac and images weren't working appropriately without this
}
);
const fargateService = new ApplicationLoadBalancedFargateService(
this,
'FooFargateService',
{
assignPublicIp: true,
cluster,
memoryLimitMiB: 1024,
cpu: 512,
desiredCount: 1,
taskImageOptions: {
containerPort: PORT,
image: containerImage
}
}
);
fargateService.targetGroup.configureHealthCheck({ path: '/health' });
const serverTable = new Table(this, 'FooTable', {
billingMode: BillingMode.PAY_PER_REQUEST,
removalPolicy: cdk.RemovalPolicy.DESTROY,
partitionKey: { name: 'id', type: AttributeType.STRING },
pointInTimeRecovery: true
});
serverTable.grantReadWriteData(fargateService.taskDefinition.taskRole);
}
}

Apparently either order in which the resources are defined matters, or the inclusion of a property from the table in the Fargate service is what did the trick. I moved the table definition up above the Fargate service and included an environment variable holding the table name and it's working as intended now.
export class FooCdkStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const vpc = new Vpc(this, 'FooVpc', {
maxAzs: 2,
natGateways: 1
});
const cluster = new Cluster(this, 'FooCluster', { vpc });
const containerImage = ContainerImage.fromAsset(
path.join(__dirname, '/../app'),
{
platform: Platform.LINUX_AMD64 // I'm on an M1 Mac and images weren't working appropriately without this
}
);
const serverTable = new Table(this, 'FooTable', {
billingMode: BillingMode.PAY_PER_REQUEST,
removalPolicy: cdk.RemovalPolicy.DESTROY,
partitionKey: { name: 'id', type: AttributeType.STRING },
pointInTimeRecovery: true
});
const fargateService = new ApplicationLoadBalancedFargateService(
this,
'FooFargateService',
{
assignPublicIp: true,
cluster,
memoryLimitMiB: 1024,
cpu: 512,
desiredCount: 1,
taskImageOptions: {
containerPort: PORT,
image: containerImage,
environment: {
SERVER_TABLE_NAME: serverTable.tableName
}
}
}
);
fargateService.targetGroup.configureHealthCheck({ path: '/health' });
serverTable.grantReadWriteData(fargateService.taskDefinition.taskRole);
}
}
Hopefully this helps someone in the future who may come across the same issue.

Related

How to add multiple base path mappings from different projects into the same AWS API Gateway?

We have separate AWS CDK projects for different APIs. We want to use same subdomain with different base path mappings for the same API Gateway resource. For example; lets say we have two APIs which are tenantApi and invoiceApi mapping to test.example.com/tenant and test.example.com/invoice. This is doable from one repository with creating one RestApi and defining multiple base path mappings to it. However, I couldn't find a way to achieve this doing it from different repositories since I need to create only one ARecord for the subdomain. My thought was creating ARecord inside a repository where we manage shared resources and importing that record from the repositories we will use the same Api Gateway.
Here is the simple aws cdk code about how I am creating an Api Gateway. As you can see, we have to pass RestApi instance into route53.RecordTarget.fromAlias so I am not really sure if we can create a ARecord before creating an Api Gateway.
export class ApiGatewayStack extends Stack {
constructor(scope: Construct, id: string, props: StackEnvProps) {
super(scope, id, props);
const tenantApi = new apigateway.RestApi(this, 'tenantApi', {
domainName: {
domainName: props.context['domainName'],
certificate: acm.Certificate.fromCertificateArn(this, 'certificateArn', props.context['certificateArn']),
basePath: 'tenant'
},
deploy: true,
deployOptions: {
stageName: 'prod',
},
defaultCorsPreflightOptions: {
allowMethods: apigateway.Cors.ALL_METHODS,
allowOrigins: apigateway.Cors.ALL_ORIGINS,
}
});
const zone = route53.HostedZone.fromLookup(this, 'Zone', { domainName: 'example.com' });
// create an alias for mapping
new route53.ARecord(this, 'domainAliasRecord', {
zone: zone,
recordName: "test",
target: route53.RecordTarget.fromAlias(new ApiGateway(tenantApi)),
});
const methodOptions: apigateway.MethodOptions = {
methodResponses: [
{
statusCode: '200',
responseParameters: {
'method.response.header.Content-Type': true,
},
},
{
statusCode: '400',
responseParameters: {
'method.response.header.Content-Type': true,
},
},
],
};
const postPaymentsLambda = new NodejsFunction(this, 'postTenantLambda', {
entry: './lambda/rest/tenant-api/post-tenant-api.ts',
handler: 'handler',
memorySize: 512,
runtime: lambda.Runtime.NODEJS_14_X,
});
// tenant/v1
const tenantV1 = tenantApi.root.addResource('v1');
tenantV1.addMethod('POST', new apigateway.LambdaIntegration(postPaymentsLambda), methodOptions);
}
}
I appreciate for any help. Thanks!
I had to first create a domainName then create a ARecord with targeting that domainName which can be imported from different APIs I want to attach.
// create domain name for api gateway
const domainName = new apigateway.DomainName(this, 'domainName', {
domainName: `test.${props.context['domainName']}`, // replace with props.type later
certificate: acm.Certificate.fromCertificateArn(this, 'certificateArn', props.context['certificateArn']),
endpointType: apigateway.EndpointType.REGIONAL,
securityPolicy: apigateway.SecurityPolicy.TLS_1_2,
});
const zone = route53.HostedZone.fromLookup(this, 'hostedZone', {
domainName: props.context['domainName']
});
// create an alias for mapping
new route53.ARecord(this, 'apiGatewayDomainAliasRecord', {
zone: zone,
recordName: 'test',
target: route53.RecordTarget.fromAlias(new r53target.ApiGatewayDomain(domainName)),
});
new CfnOutput(this, 'apiGatewayDomainNameAliasTarget', {
value: domainName.domainNameAliasDomainName,
description: 'domainNameAliasTarget attribute used when importing domain name',
exportName: 'apiGatewayDomainNameAliasTarget'
});
Later on, I will import this domainName to create a BasePathMapping. There are three attributes used when importing a domainName;
domainName: the domainName we created before.
domainNameAliasHostedZoneId: Hosted ZoneId where the domain is defined.
domainNameAliasTarget: AWS documentation doesn't clearly state out about what it is. Basically, it is the domainNameAliasDomainName value of the domainName we created at the very first place.
const tenantApi = new apigateway.RestApi(this, 'tenantApi', {
deployOptions: {
stageName: 'dev',
},
deploy: true,
defaultCorsPreflightOptions: {
allowMethods: apigateway.Cors.ALL_METHODS,
allowOrigins: apigateway.Cors.ALL_ORIGINS,
}
});
const domainName = apigateway.DomainName.fromDomainNameAttributes(this, 'domainName', {
domainName: `test.${props.context['domainName']}`,
domainNameAliasHostedZoneId: props.context['hostedZoneId'],
domainNameAliasTarget: Fn.importValue(props.context['domainNameAliasTarget']),
});
const nodeBasePathMapping = new apigateway.BasePathMapping(this, 'nodeBasePathMapping', {
basePath: 'node',
domainName,
restApi: tenantApi,
});

ECS task unable to pull secrets or registry auth

I have a CDK project that creates a CodePipeline which deploys an application on ECS. I had it all previously working, but the VPC was using a NAT gateway, which ended up being too expensive. So now I am trying to recreate the project without requiring a NAT gateway. I am almost there, but I have now run into issues when the ECS service is trying to start tasks. All tasks fail to start with the following error:
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 5 time(s): failed to fetch secret
At this point I've kind of lost track of the different things I have tried, but I will post the relevant bits here as well as some of my attempts.
const repository = ECR.Repository.fromRepositoryAttributes(
this,
"ecr-repository",
{
repositoryArn: props.repository.arn,
repositoryName: props.repository.name,
}
);
// vpc
const vpc = new EC2.Vpc(this, this.resourceName(props, "vpc"), {
maxAzs: 2,
natGateways: 0,
enableDnsSupport: true,
});
const vpcSecurityGroup = new SecurityGroup(this, "vpc-security-group", {
vpc: vpc,
allowAllOutbound: true,
});
// tried this to allow the task to access secrets manager
const vpcEndpoint = new EC2.InterfaceVpcEndpoint(this, "secrets-manager-task-vpc-endpoint", {
vpc: vpc,
service: EC2.InterfaceVpcEndpointAwsService.SSM,
});
const secrets = SecretsManager.Secret.fromSecretCompleteArn(
this,
"secrets",
props.secrets.arn
);
const cluster = new ECS.Cluster(this, this.resourceName(props, "cluster"), {
vpc: vpc,
clusterName: `api-cluster`,
});
const ecsService = new EcsPatterns.ApplicationLoadBalancedFargateService(
this,
"ecs-service",
{
taskSubnets: {
subnetType: SubnetType.PUBLIC,
},
securityGroups: [vpcSecurityGroup],
serviceName: "api-service",
cluster: cluster,
cpu: 256,
desiredCount: props.scaling.desiredCount,
taskImageOptions: {
image: ECS.ContainerImage.fromEcrRepository(
repository,
this.ecrTagNameParameter.stringValue
),
secrets: getApplicationSecrets(secrets), // returns
logDriver: LogDriver.awsLogs({
streamPrefix: "api",
logGroup: new LogGroup(this, "ecs-task-log-group", {
logGroupName: `${props.environment}-api`,
}),
logRetention: RetentionDays.TWO_MONTHS,
}),
},
memoryLimitMiB: 512,
publicLoadBalancer: true,
domainZone: this.hostedZone,
certificate: this.certificate,
redirectHTTP: true,
}
);
const scalableTarget = ecsService.service.autoScaleTaskCount({
minCapacity: props.scaling.desiredCount,
maxCapacity: props.scaling.maxCount,
});
scalableTarget.scaleOnCpuUtilization("cpu-scaling", {
targetUtilizationPercent: props.scaling.cpuPercentage,
});
scalableTarget.scaleOnMemoryUtilization("memory-scaling", {
targetUtilizationPercent: props.scaling.memoryPercentage,
});
secrets.grantRead(ecsService.taskDefinition.taskRole);
repository.grantPull(ecsService.taskDefinition.taskRole);
I read somewhere that it probably has something to do with Fargate version 1.4.0 vs 1.3.0, but I'm not sure what I need to change to allow the tasks to access what they need to run.
You need to create an interface endpoints for Secrets Manager, ECR (two types of endpoints), CloudWatch, as well as a gateway endpoint for S3.
Refer to the documentation on the topic.
Here's an example in Python, it'd work the same in TS:
vpc.add_interface_endpoint(
"secretsmanager_endpoint",
service=ec2.InterfaceVpcEndpointAwsService.SECRETS_MANAGER,
)
vpc.add_interface_endpoint(
"ecr_docker_endpoint",
service=ec2.InterfaceVpcEndpointAwsService.ECR_DOCKER,
)
vpc.add_interface_endpoint(
"ecr_endpoint",
service=ec2.InterfaceVpcEndpointAwsService.ECR,
)
vpc.add_interface_endpoint(
"cloudwatch_logs_endpoint",
service=ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS,
)
vpc.add_gateway_endpoint(
"s3_endpoint",
service=ec2.GatewayVpcEndpointAwsService.S3
)
Keep in mind that interface endpoints cost money as well, and may not be cheaper than a NAT.

AWS CDK ignores taskSubnets selection

I have a flask app which I'm trying to deploy into a landing zone, using CDK (Typescript, v2.5.0) as a Fargate instance.
The landing zone an existing VPC which I need to use, with Isolated and Private subnets.
I've tried every combination I can think of to get the load balancer (tried both application- and network-balanced) to use the Isolated subnets, but nothing has worked.
The error I get from cdk synth is
deploy/node_modules/aws-cdk-lib/aws-ec2/lib/vpc.ts:401
throw new Error(`There are no '${subnetType}' subnet groups in this VPC. Available types: ${availableTypes}`);
^
*Error: There are no 'Public' subnet groups in this VPC. Available types: Isolated*
Here's my code:
import * as cdk from "#aws-cdk/core";
import { Stack, StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as ecs from "aws-cdk-lib/aws-ecs";
import * as ecsp from "aws-cdk-lib/aws-ecs-patterns";
export class DeployStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const defaultnonprodVPC = "existing-vpc";
const defaultVPC = ec2.Vpc.fromLookup(this,
"defaultVPC",
{
isDefault: false,
vpcId: defaultnonprodVPC,
tags: { "aws-cdk:subnet-type": "isolated" }
}
);
const knownIsolatedSubnets = defaultVPC.isolatedSubnets;
const monitoringSubnets = defaultVPC.selectSubnets(
{
subnetType: ec2.SubnetType.PRIVATE_ISOLATED
}
);
const networkBalancedFargateService = new ecsp.NetworkLoadBalancedFargateService(this,
"ConnectorMonitorService", {
memoryLimitMiB: 512,
desiredCount: 1,
cpu: 256,
taskImageOptions: {
image: ecs.ContainerImage.fromAsset("../src")
},
taskSubnets:
{
subnetType: ec2.SubnetType.PRIVATE_ISOLATED
},
vpc: defaultVPC
});
}
}
Changing the taskSubnets to any of
{ subnets: { knownIsolatedSubnets } }
or
subnetGroupName: "subnet-existing-subnet-name"
or
monitoringSubnets
makes no difference to cdk synth. Setting assignPublicIp: false doesn't change things either.
What am I doing wrong, or missing?
NetworkLoadBalancedFargateService has a property publicLoadBalancer which is true by default. This makes the Load Balancer internet-facing, which is not correct in your case. You need to set it to false so that private or isolated subnets work.
Documentation for NetworkLoadBalancedFargateService

How to provide a custom domain name for a Lambda-based APIGateway in CDK?

For the purposes of this question, assume that I already have an example.org Hosted Zone in Route53 (my actual zone is, of course, different)
With the following CDK app:
export class MyExampleStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const backend = new Function(this, 'backendLambda', {
code: new AssetCode("lambda/"),
handler: "index.handler",
runtime: Runtime.PYTHON_3_8
});
apiDomainName = 'api.test.example.org'
const api = new LambdaRestApi(this, 'api', {
handler: backend,
proxy: true,
deploy: true,
domainName: {
domainName: apiDomainName,
certificate: new Certificate(this, 'apiCertificate', {
domainName: apiDomainName
})
}
});
}
}
, when I run cdk deploy, part of the output reads:
Outputs:
MyExampleStack.apiEndpoint0F54D2EA = https://<alphanumericId>.execute-api.us-east-1.amazonaws.com/prod/
, and, indeed, when I curl that url, I see the response I would expect from my Lambda code. I would expect curling api.test.example.org to give the same result - however, instead it gives curl: (6) Could not resolve host: api.test.example.org.
Based on this documentation, I tried:
export class MyExampleStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const rootDomain = 'example.org'
const zone = HostedZone.fromLookup(this, 'baseZone', {
domainName: rootDomain
});
const backend = new Function(...);
const api = new LambdaRestApi(...);
new ARecord(this, 'apiDNS', {
zone: zone,
recordName: 'api.test',
target: RecordTarget.fromAlias(new ApiGateway(api))
});
}
}
which did give a Route53 entry:
$ aws route53 list-hosted-zones
{
"HostedZones": [
{
"Id": "/hostedzone/ZO3B2N6W70PDD",
"Name": "example.org.",
"CallerReference": "598D71AB-4A98-EC5A-A170-D51CB243A2EA",
"Config": {
"PrivateZone": false
},
"ResourceRecordSetCount": 8
}
]
}
$ aws route53 list-resource-record-sets --hosted-zone-id /hostedzone/ZO3B2N6W70PDD --query 'ResourceRecordSets[?Name==`api.test.example.org.`]'
[
{
"Name": "api.test.example.org.",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "Z1UJRXOUMOOFQ9",
"DNSName": "<alphanumericId2>.execute-api.us-east-1.amazonaws.com.",
"EvaluateTargetHealth": false
}
}
]
But this still doesn't work:
curl api.test.example.org still gives "Could not resolve host"
curl <alphanumericId2>.execute-api.us-east-1.amazonaws.com gives curl: (7) Failed to connect to <alphanumericId2>.execute-api.us-east-1.amazonaws.com port 80: Connection refused
curl https://<alphanumericId2>..execute-api.us-east-1.amazonaws.com gives {"message":"Forbidden"}
curl https://<alphanumericId>.[...] (i.e. the output from cdk deploy) still gives the expected response from the Lambda
How can I define a custom name in Route53 to route to my Lambda-backed APIGateway API?
Overall code LambdaRestApi with Route53 A Record, will create
Custom domain pointing to a particular stage prod i.e api.test.example.org domain to stage `prod'(example)
Route 53 A record for api.test.example.org pointing to Api Gateway hosted zone.
These are two combinations that will work
https://api.test.example.org will work pointing directly to stage prod.
CDK Output https://abcdef1234.execute-api.us-east-1.amazonaws.com/prod/ will work as stage is appended to it.
These are few combinations that will not work
Two other tests you did with http://
With no protocol, defaults to http, will not work, as we api gateway by default gives TLS 1.0 (ssl-https) and no http listener.
One other attempt you did with https:// without a stage name at the end, will return 403 forbidden, as stage name is missing.
Here is full CDK code.
import * as cdk from "#aws-cdk/core";
import * as apigw from "#aws-cdk/aws-apigateway";
import * as acm from "#aws-cdk/aws-certificatemanager";
import * as route53 from "#aws-cdk/aws-route53";
import * as route53Targets from "#aws-cdk/aws-route53-targets";
import * as lambda from "#aws-cdk/aws-lambda";
export class HelloCdkStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
this.buildLambdaApiGateway();
}
buildLambdaApiGateway() {
const rootDomain = "example.org";
const zone = route53.HostedZone.fromLookup(this, "baseZone", {
domainName: rootDomain,
});
const backend = new lambda.Function(this, "MyLayeredLambda", {
code: new lambda.InlineCode("foo"),
handler: "index.handler",
runtime: lambda.Runtime.NODEJS_10_X,
});
const restApi = new apigw.LambdaRestApi(this, "myapi", {
handler: backend,
domainName: {
domainName: `api-test.${rootDomain}`,
certificate: acm.Certificate.fromCertificateArn(
this,
"my-cert",
"arn:aws:acm:us-east-1:111112222333:certificate/abcd6805-1234-4159-ac38-761acdc700ef"
),
endpointType: apigw.EndpointType.REGIONAL,
},
});
new route53.ARecord(this, "apiDNS", {
zone: zone,
recordName: "api-test",
target: route53.RecordTarget.fromAlias(
new route53Targets.ApiGateway(restApi)
),
});
}
}

AWS CDK destroy fails to delete secret

I have a CDK script that creates an S3 bucket, VPC, and an RDS instance. Deploy is working, but the destroy fails with an error that my user is not authorized to secretsmanager:DeleteSecret.
I used the IAM policy testing tool to check and it passes. I am able to delete the secret via the UI. The CDK destroy command continues to fail though. Any thoughts?
CDK script:
class AcmeCdkStack extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// create a general purpose bucket for use with the app
new s3.Bucket(this, 'app-bucket', {
versioned: true
});
// create a vpc for our application
const vpc = new ec2.Vpc(this, 'app-vpc, {
cidr: "10.0.0.0/16",
});
// create a database instance
const db = new rds.DatabaseInstance(this, `app-db`, {
engine: rds.DatabaseInstanceEngine.POSTGRES,
instanceClass: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.MICRO),
vpc,
masterUsername: `dbadmin`,
deleteAutomatedBackups: false,
deletionProtection: false,
// https://github.com/aws/aws-cdk/issues/4036
removalPolicy: cdk.RemovalPolicy.DESTROY,
});
}
}
const app = new cdk.App();
new AcmeCdkStack(app, 'app-stack;);
Error:
User: arn:aws:iam::0000000000:user/user#acme.com is not authorized to perform: secretsmanager:DeleteSecret on resource: arn:aws:secretsmanager:us-east-1:0000000000:secret:appdbdemoSecret0261-mjgIXOsp5rLL-HxFng1 (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException; Request ID: 000000000)
Based on the comments, the problem was that the CDK was using different credentials than expected. The solution was to use correct AWS_PROFILE.