CDK to enable DNS resolution for VPCPeering - amazon-web-services

I have VPC peering to connect to a lambda in one aws account to a RDS instance in another aws account. This works fine but required the VPC peering to have DNS resolution option enabled.
By default DNS resolution is set to :
DNS resolution from accepter VPC to private IP :Disabled.
This can be done via the AWS console and the CLI. I am not able to achieve the same using AWS CDK.
https://docs.aws.amazon.com/vpc/latest/peering/modify-peering-connections.html
The CfnVPCPeeringConnection does not seem to have this option.
https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-ec2.CfnVPCPeeringConnection.html
Is there any other way of achieving this via CDK ?
const cfnVPCPeeringConnection :CfnVPCPeeringConnection =
new CfnVPCPeeringConnection(
stack,
"vpcPeeringId",
{
peerVpcId : "<vpcId of acceptor account>",
vpcId : "<reference of the Id>",
peerOwnerId : "<aws acc number>",
peerRegion : "<region>",
peerRoleArn :"<arn created in the acceptor account>"",
}
);
//update route tables
rdsConnectorVpc.isolatedSubnets.forEach(({ routeTable: { routeTableId } }, index) => {
new CfnRoute(this.parentStack, 'PrivateSubnetPeeringConnectionRoute' + index, {
destinationCidrBlock: '<CIDR>',
routeTableId,
vpcPeeringConnectionId: cfnVPCPeeringConnection.ref,
})
});

You can use a CustomResource Construct in AWS CDK to achieve it:
import * as cdk from "#aws-cdk/core";
import ec2 = require("#aws-cdk/aws-ec2");
import iam = require("#aws-cdk/aws-iam");
import { AwsCustomResource, AwsCustomResourcePolicy, AwsSdkCall, PhysicalResourceId } from "#aws-cdk/custom-resources";
import { RetentionDays } from "#aws-cdk/aws-logs";
export interface AllowVPCPeeringDNSResolutionProps {
vpcPeering: ec2.CfnVPCPeeringConnection,
}
export class AllowVPCPeeringDNSResolution extends cdk.Construct {
constructor(scope: cdk.Construct, id: string, props: AllowVPCPeeringDNSResolutionProps) {
super(scope, id);
const onCreate:AwsSdkCall = {
service: "EC2",
action: "modifyVpcPeeringConnectionOptions",
parameters: {
VpcPeeringConnectionId: props.vpcPeering.ref,
AccepterPeeringConnectionOptions: {
AllowDnsResolutionFromRemoteVpc: true,
},
RequesterPeeringConnectionOptions: {
AllowDnsResolutionFromRemoteVpc: true
}
},
physicalResourceId: PhysicalResourceId.of(`allowVPCPeeringDNSResolution:${props.vpcPeering.ref}`)
};
const onUpdate = onCreate;
const onDelete:AwsSdkCall = {
service: "EC2",
action: "modifyVpcPeeringConnectionOptions",
parameters: {
VpcPeeringConnectionId: props.vpcPeering.ref,
AccepterPeeringConnectionOptions: {
AllowDnsResolutionFromRemoteVpc: false,
},
RequesterPeeringConnectionOptions: {
AllowDnsResolutionFromRemoteVpc: false
}
},
};
const customResource = new AwsCustomResource(this, "allow-peering-dns-resolution", {
policy: AwsCustomResourcePolicy.fromStatements([
new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
resources: ["*"],
actions: [
"ec2:ModifyVpcPeeringConnectionOptions",
]
}),
]),
logRetention: RetentionDays.ONE_DAY,
onCreate,
onUpdate,
onDelete,
});
customResource.node.addDependency(props.vpcPeering);
}
}
and use it like this:
[...]
const peerConnection = new ec2.CfnVPCPeeringConnection(this, "peerConnection", {
vpcId: destinationVPC.vpcId,
peerVpcId: lambdaVPCToDestinationVPC.vpcId,
});
new AllowVPCPeeringDNSResolution(this, "peerConnectionDNSResolution", {
vpcPeering: peerConnection,
});
[...]

Related

How to grant access to a private load-balancer from an EBS environment using CDK?

I have an existing elasticbeanstalk environment, say hello-env. And I have created an ApplicationLoadBalancedFargateService. I have made this service private. But I want to allow hello-env to access the load-balancer URL. How do I achieve that using CDK?
const cdk = require("aws-cdk-lib");
const { Construct } = require("constructs");
const ecs = require("aws-cdk-lib/aws-ecs");
const ecsp = require("aws-cdk-lib/aws-ecs-patterns");
const ecr = require("aws-cdk-lib/aws-ecr");
const kms = require("aws-cdk-lib/aws-kms");
const ebs = require("aws-cdk-lib/aws-elasticbeanstalk");
class MyEcsStack extends cdk.Stack {
constructor(scope, id, props) {
super(scope, id, props);
const myPvtService = new ecsp.ApplicationLoadBalancedFargateService(
this,
"MyPvtService",
{
cpu: 256,
memoryLimitMiB: 512,
desiredCount: 1,
taskImageOptions: {
image: ecs.ContainerImage.fromEcrRepository(
ecr.Repository.fromRepositoryAttributes(this, "MyPvtApp", {
repositoryArn:
"arn:aws:ecr:us-east-1:<my-account>:repository/my-pvt-app",
repositoryName: "my-pvt-app",
}),
"dev"
),
containerPort: 9000,
},
publicLoadBalancer: false,
}
);
const myKey = kms.Key.fromKeyArn(
this,
"myKey",
"arn:aws:kms:us-east-1:<my-account>:key/mrk-28490388455e548c1998f64c89f853f61"
);
myKey.grant(
myPvtService.taskDefinition.taskRole,
"kms:GetPublicKey"
);
myKey.grant(myPvtService.taskDefinition.taskRole, "kms:Sign");
}
}
module.exports = { MyEcsStack };
In the above code, I am allowing myPvtService to access the KMS keys. Similarly I want to allow an EBS Environment to access the load balancer service. How do I achieve that? There is no way to create and EBS env object using something like fromArn.

Got Error "Either integrationSubtype` or `integrationUri` must be specified" when try to integration Fargate service with API gateway

I'm trying to create a public API which will be integrated with a Fargate service which already exists in private subnet.
I got below error when run cdk synthesize --profile=PandaService-Alpha.
/Users/yangliu/Projects/Panda/PandaApi/node_modules/#aws-cdk/aws-apigatewayv2-alpha/lib/http/integration.ts:249
throw new Error('Either `integrationSubtype` or `integrationUri` must be specified.');
^
Error: Either `integrationSubtype` or `integrationUri` must be specified.
at new HttpIntegration (/Users/yangliu/Projects/Panda/PandaApi/node_modules/#aws-cdk/aws-apigatewayv2-alpha/lib/http/integration.ts:249:13)
at HttpAlbIntegration._bindToRoute (/Users/yangliu/Projects/Panda/PandaApi/node_modules/#aws-cdk/aws-apigatewayv2-alpha/lib/http/integration.ts:317:26)
at new HttpRoute (/Users/yangliu/Projects/Panda/PandaApi/node_modules/#aws-cdk/aws-apigatewayv2-alpha/lib/http/route.ts:191:38)
at /Users/yangliu/Projects/Panda/PandaApi/node_modules/#aws-cdk/aws-apigatewayv2-alpha/lib/http/api.ts:458:14
at Array.map (<anonymous>)
at HttpApi.addRoutes (/Users/yangliu/Projects/Panda/PandaApi/node_modules/#aws-cdk/aws-apigatewayv2-alpha/lib/http/api.ts:455:20)
at ApigatewayStack.addApiRoutes (/Users/yangliu/Projects/Panda/PandaApi/lib/apigateway-stack.ts:110:22)
at new ApigatewayStack (/Users/yangliu/Projects/Panda/PandaApi/lib/apigateway-stack.ts:101:10)
at /Users/yangliu/Projects/Panda/PandaApi/bin/app.ts:17:3
The error is thrown in the addApiRoutes method in below code.
Code
import * as CDK from "aws-cdk-lib";
import * as CertificateManager from "aws-cdk-lib/aws-certificatemanager";
import * as Route53 from "aws-cdk-lib/aws-route53";
import * as ApiGatewayV2Alpha from "#aws-cdk/aws-apigatewayv2-alpha";
import * as ApiGatewayV2IntegrationsAlpha from "#aws-cdk/aws-apigatewayv2-integrations-alpha";
import * as ELBv2 from "aws-cdk-lib/aws-elasticloadbalancingv2";
import { Construct } from "constructs";
import { StageInfo } from "../config/stage-config";
import * as EC2 from "aws-cdk-lib/aws-ec2";
export interface ApigatewayStackProps extends CDK.StackProps {
readonly packageName: string;
readonly stageInfo: StageInfo;
}
export class ApigatewayStack extends CDK.Stack {
private readonly coreVpc: EC2.IVpc;
// Prefix for CDK constrcut ID
private readonly constructIdPrefix: string;
private readonly pandaApi: ApiGatewayV2Alpha.HttpApi;
constructor(scope: Construct, id: string, props: ApigatewayStackProps) {
super(scope, id, props);
this.coreVpc = EC2.Vpc.fromLookup(
this,
`${props.stageInfo.stageName}VpcLookupId`,
{
vpcName: "CoreVpc",
}
);
this.constructIdPrefix = `${props.packageName}-${props.stageInfo.stageName}`;
const hostedZone: Route53.IHostedZone = Route53.HostedZone.fromLookup(
this,
`${this.constructIdPrefix}-HostedZoneLookup`,
{
domainName: props.stageInfo.domainName,
}
);
const domainCertificate = new CertificateManager.Certificate(
this,
`${this.constructIdPrefix}-pandaApiCertificate`,
{
domainName: props.stageInfo.domainName,
validation:
CertificateManager.CertificateValidation.fromDns(hostedZone),
}
);
const customDomainName = new ApiGatewayV2Alpha.DomainName(
this,
`${this.constructIdPrefix}-ApiGatewayDomainName`,
{
certificate: domainCertificate,
domainName: props.stageInfo.domainName,
}
);
this.pandaApi = new ApiGatewayV2Alpha.HttpApi(
this,
`${this.constructIdPrefix}-pandaApi`,
{
defaultDomainMapping: {
domainName: customDomainName,
//mappingKey: props.pipelineStageInfo.stageName
},
corsPreflight: {
allowOrigins: ["*"],
allowHeaders: ["*"],
allowMethods: [
ApiGatewayV2Alpha.CorsHttpMethod.OPTIONS,
ApiGatewayV2Alpha.CorsHttpMethod.GET,
ApiGatewayV2Alpha.CorsHttpMethod.POST,
ApiGatewayV2Alpha.CorsHttpMethod.PUT,
],
maxAge: CDK.Duration.hours(6),
},
//createDefaultStage: false,
// only allow use custom domain
disableExecuteApiEndpoint: true
}
);
this.addApiRoutes(props);
}
/**
* Add API routes for multiple services.
*/
private addApiRoutes(props: ApigatewayStackProps) {
const PandaServiceIntegration : ApiGatewayV2IntegrationsAlpha.HttpAlbIntegration =
this.generatePandaServiceIntegration(props);
this.pandaApi.addRoutes({
path: "/products",
methods: [ApiGatewayV2Alpha.HttpMethod.ANY],
integration: PandaServiceIntegration,
});
this.pandaApi.addRoutes({
path: "/store-categories",
methods: [ApiGatewayV2Alpha.HttpMethod.ANY],
integration: PandaServiceIntegration,
});
this.pandaApi.addRoutes({
path: "/stores",
methods: [ApiGatewayV2Alpha.HttpMethod.ANY],
integration: PandaServiceIntegration,
});
}
/**
*
* #returns HttpAlbIntegration for PandaService.
*/
private generatePandaServiceIntegration(props: ApigatewayStackProps) {
const vpcLink = new ApiGatewayV2Alpha.VpcLink(
this,
`${this.constructIdPrefix}-VpcLink`,
{
vpc: this.coreVpc,
subnets: {
subnetType: EC2.SubnetType.PRIVATE_ISOLATED,
},
}
);
const PandaServiceAlbSecurityGroup = EC2.SecurityGroup.fromLookupByName(
this,
`${this.constructIdPrefix}-PandaServiceAlbSecurityGroupLookup`,
"PandaServiceAlbSecurityGroup",
this.coreVpc
);
const PandaServiceAlbListener : ELBv2.IApplicationListener =
ELBv2.ApplicationListener.fromApplicationListenerAttributes(this, `${this.constructIdPrefix}-PandaServiceAlbListenerLookUp`, {
listenerArn: props.stageInfo.PandaServiceAlbArn,
securityGroup: PandaServiceAlbSecurityGroup,
});
const PandaServiceIntegration: ApiGatewayV2IntegrationsAlpha.HttpAlbIntegration =
new ApiGatewayV2IntegrationsAlpha.HttpAlbIntegration(
`${this.constructIdPrefix}-PandaServiceIntegration`,
PandaServiceAlbListener ,
{
method: ApiGatewayV2Alpha.HttpMethod.ANY,
vpcLink: vpcLink,
secureServerName: props.stageInfo.domainName,
parameterMapping: new ApiGatewayV2Alpha.ParameterMapping()
}
);
return PandaServiceIntegration;
}
}
As Otavio pointed out, my props.stageInfo.PandaServiceAlbArn is an empty string, after updating it with the actual string the problem get resolved.

Can we test whether Lambda is connected with correct subnets using AWS CDK assertions, Jest during the runtime synth?

I have a global stack, where I have my predefined VPCs and subnets
export class GlobalStack extends InternalStack {
/**
* Shared VPC Instance
*/
public readonly vpc = Vpc.fromLookup(this, 'vpc', {
vpcId: process.env.VPC_ID!,
})
/**
* Private Subnet 1
*/
public privateSubnet1 = new Subnet(this, 'subnet-1', {
vpcId: this.vpc.vpcId,
cidrBlock: 'xx.x.x.x/xx',
availabilityZone: `${this.region}-1`,
})
/**
* Private Subnet 2
*/
public privateSubnet2 = new Subnet(this, 'subnet-2', {
vpcId: this.vpc.vpcId,
cidrBlock: 'xx.x.x.x/xx',
availabilityZone: `${this.region}-2`,
})
/**
* Private Subnet 3
*/
public privateSubnet3 = new Subnet(this, 'subnet-3', {
vpcId: this.vpc.vpcId,
cidrBlock: 'xx.x.x.x/xx',
availabilityZone: `${this.region}-3`,
})
public readonly apiGatewayVpcEndpoint = this.vpc.addInterfaceEndpoint(
'ApiGateway',
{
service: InterfaceVpcEndpointAwsService.APIGATEWAY,
subnets: {
subnets: [
this.privateSubnet1,
this.privateSubnet2,
this.privateSubnet3,
],
},
}
)
}
And Once I get these vpc and subnet , I pass them to our lambda , which we have an enhanced node js lambda (a wrapper around node js lambda function), where I check if vpc props are available, I will add the defined three subnets.
The constructor of my lambda function:
constructor(scope: Construct, id: string, props: EnhancedNodeJsLambdaProps) {
super(scope, id, {
...props,
...(props.vpc && {
vpcSubnets: props.vpc.selectSubnets({
subnetFilters: [
SubnetFilter.containsIpAddresses(['xx.x.x.x/xx', 'xx.x.x.x/xx', 'xx.x.x.x/xx']
),
],
}),
}),
runtime: props.runtime || Runtime.NODEJS_12_X,
tracing: Tracing.ACTIVE,
})
}
So now when I try to test my lambda, whether the subnets are attached to it, I either get some dummy az values or it returns that the lambda isn't connected with subnets, how can I test the same?
FYR, I will attach some of my test cases below
it('testing vpc subnets ', async () => {
const app = new cdk.App()
const topicsStack = new cdk.Stack(app, 'TopicsStack')
const globalStack = await new GlobalStack(app, 'global-stack', {
stackName: 'global-stack',
description: 'Global Resources (Shared at the account level)',
env: {
account: '11111111',
region: 'us-east-1',
},
envName: 'test',
})
let newLambda = new EnhancedNodeJsLambda(topicsStack, 'test-lambda', {
entry,
connectionReuseEnabled: true,
vpc: globalStack.vpc,
})
console.log(
globalStack.vpc.selectSubnets({
subnetFilters: [
SubnetFilter.containsIpAddresses(['xx.x.x.x/xx', 'xx.x.x.x/xx', 'xx.x.x.x/xx']),
],
}).availabilityZones
)
//how to test subnets are properly linked?
})
Also about the dummy values, the console log in the test case above returns me some dummy az values instead of my code ones:
the console log returns
[ 'dummy1a', 'dummy1b' ]
I tried to test the connection by adding the below code to the test case above
const othertemp = Template.fromStack(topicsStack)
othertemp.hasResourceProperties('AWS::Lambda::Function', {
VpcConfig: {
SubnetIds: globalStack.vpc.selectSubnets({
subnetFilters: [
SubnetFilter.containsIpAddresses(['xx.x.x.x/xx', 'xx.x.x.x/xx', 'xx.x.x.x/xx']
),
],
}),
},
})
But it failed saying
with the following mismatches:
Expected type object but received array at /Properties/VpcConfig/SubnetIds (using objectLike matcher)
Also the subnet id's and az's are dummy values and not my intended ones.
I'm not sure why it's returning dummy values instead of my pre-defined one, and also I don't know how to test the lambda is connected with the correct subnets or no.
Well, not sure how to do it with CDK assertions, but I was able to get it done with jest assertions:
expect(myFunction.Properties.VpcConfig.SubnetIds).toEqual('my-subnet-ids');

Pulumi GCP MemoryStore Redis Cache Internal Server Error 13

I have a weird scenario here.
The following line in my Pulumi typescript code always fails the first time:
const redisCache = new gcp.redis.Instance("my-redis-cache", {
name: "my-metadata-cache",
tier: "BASIC",
memorySizeGb: 1,
authorizedNetwork: pulumi.interpolate`projects/someprojectid/global/networks/default`,
connectMode: "PRIVATE_SERVICE_ACCESS",
redisVersion: "REDIS_6_X",
displayName: "My Metadata Cache",
project: someprojectid,
}, defaultResourceOptions);
**
error: 1 error occurred:
* Error waiting to create Instance: Error waiting for Creating Instance: Error code 13, message: an internal error has occurred
**
Strangely, when I again run pulumi up, it succeeds. Has anyone else faced this before? Any clues?
Ok this turned out to be a case of working with a beast of a code. Once I started isolating the issue, things became clearer. For those who stumble across this one, here is a full working code.
import * as pulumi from "#pulumi/pulumi";
import * as gcp from "#pulumi/gcp";
export interface CacheComponentResourceArgs {
projectId : pulumi.Input<string>;
projectNumber: pulumi.Input<string>;
}
export class CacheComponentResource extends pulumi.ComponentResource {
constructor(name: string, resourceArgs: CacheComponentResourceArgs, opts?: pulumi.ResourceOptions) {
const inputs: pulumi.Inputs = {
options: opts,
};
super("ekahaa:abstracta:Cache", name, inputs, opts);
const serviceNetworkingAccessService = new gcp.projects.Service("service-nw-" + name , {
disableDependentServices: true,
project: resourceArgs.projectId,
service: "servicenetworking.googleapis.com",
}, {
parent : this
});
const redisService = new gcp.projects.Service("redis-service-" + name, {
disableDependentServices: true,
project: resourceArgs.projectId,
service: "redis.googleapis.com",
}, {
parent : this
});
const defaultGlobalAddress = new gcp.compute.GlobalAddress("default-ip-range-" + name, {
name: "default-ip-range",
purpose: "VPC_PEERING",
prefixLength: 16,
project: resourceArgs.projectId,
addressType: "INTERNAL",
network: pulumi.interpolate`projects/${resourceArgs.projectId}/global/networks/default`
}, {
parent : this,
dependsOn: [ redisService]
});
const privateServiceConnection = new gcp.servicenetworking.Connection("servicenetworking-" + name, {
service: "servicenetworking.googleapis.com",
network: pulumi.interpolate`projects/${resourceArgs.projectId}/global/networks/default`,
reservedPeeringRanges: [defaultGlobalAddress.name],
}, {
parent : this,
dependsOn: [ defaultGlobalAddress]
});
const iamBindingRedis2 = new gcp.projects.IAMBinding("iamredis2-" + name, {
members: [
pulumi.interpolate`serviceAccount:service-${resourceArgs.projectNumber}#service-networking.iam.gserviceaccount.com`
],
role: "roles/servicenetworking.serviceAgent",
project: resourceArgs.projectId
}, {
parent : this,
dependsOn: [privateServiceConnection]
});
const redisCache = new gcp.redis.Instance(name, {
name: name,
tier: "BASIC",
memorySizeGb: 1,
authorizedNetwork: pulumi.interpolate`projects/${resourceArgs.projectId}/global/networks/default`,
connectMode: "PRIVATE_SERVICE_ACCESS",
redisVersion: "REDIS_6_X",
displayName: "Abstracta Metadata Cache",
project: resourceArgs.projectId,
}, {
parent : this,
dependsOn : [redisService,serviceNetworkingAccessService,iamBindingRedis2]
});
this.registerOutputs({
redisCache : redisCache
});
}
}
let suffix = "20211018-002";
let org_name = `org-redis-demo-${suffix}`;
let projectId = `redis-demo-${suffix}` ;
const myGcpProject = new gcp.organizations.Project('ab-' + org_name, {
orgId: gcpOrgId,
projectId: projectId,
billingAccount: billingAccountId,
name: 'ab-' + org_name,
});
const myGcpProjectIAM = new gcp.projects.IAMBinding("iam-001", {
members: [
"user:vikram.vasudevan#ekahaa.com",
],
role: "roles/owner",
project: myGcpProject.projectId
});
const cacheComponentResource = new CacheComponentResource("my-cache", {
projectId : myGcpProject.projectId,
projectNumber : myGcpProject.number
}, {
dependsOn : [myGcpProjectIAM]
});

AWS CDK - How to add an event notification to an existing S3 Bucket

I'm trying to modify this AWS-provided CDK example to instead use an existing bucket. Additional documentation indicates that importing existing resources is supported. So far I am unable to add an event notification to the existing bucket using CDK.
Here is my modified version of the example:
class S3TriggerStack(core.Stack):
def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
super().__init__(scope, id, **kwargs)
# create lambda function
function = _lambda.Function(self, "lambda_function",
runtime=_lambda.Runtime.PYTHON_3_7,
handler="lambda-handler.main",
code=_lambda.Code.asset("./lambda"))
# **MODIFIED TO GET EXISTING BUCKET**
#s3 = _s3.Bucket(self, "s3bucket")
s3 = _s3.Bucket.from_bucket_arn(self, 's3_bucket',
bucket_arn='arn:<my_region>:::<my_bucket>')
# create s3 notification for lambda function
notification = aws_s3_notifications.LambdaDestination(function)
# assign notification for the s3 event type (ex: OBJECT_CREATED)
s3.add_event_notification(_s3.EventType.OBJECT_CREATED, notification)
This results in the following error when trying to add_event_notification:
AttributeError: '_IBucketProxy' object has no attribute 'add_event_notification'
The from_bucket_arn function returns an IBucket, and the add_event_notification function is a method of the Bucket class, but I can't seem to find any other way to do this. Maybe it's not supported. Any help would be appreciated.
I managed to get this working with a custom resource. It's TypeScript, but it should be easily translated to Python:
const uploadBucket = s3.Bucket.fromBucketName(this, 'BucketByName', 'existing-bucket');
const fn = new lambda.Function(this, 'MyFunction', {
runtime: lambda.Runtime.NODEJS_10_X,
handler: 'index.handler',
code: lambda.Code.fromAsset(path.join(__dirname, 'lambda-handler'))
});
const rsrc = new AwsCustomResource(this, 'S3NotificationResource', {
onCreate: {
service: 'S3',
action: 'putBucketNotificationConfiguration',
parameters: {
// This bucket must be in the same region you are deploying to
Bucket: uploadBucket.bucketName,
NotificationConfiguration: {
LambdaFunctionConfigurations: [
{
Events: ['s3:ObjectCreated:*'],
LambdaFunctionArn: fn.functionArn,
Filter: {
Key: {
FilterRules: [{ Name: 'suffix', Value: 'csv' }]
}
}
}
]
}
},
// Always update physical ID so function gets executed
physicalResourceId: 'S3NotifCustomResource' + Date.now().toString()
}
});
fn.addPermission('AllowS3Invocation', {
action: 'lambda:InvokeFunction',
principal: new iam.ServicePrincipal('s3.amazonaws.com'),
sourceArn: uploadBucket.bucketArn
});
rsrc.node.addDependency(fn.permissionsNode.findChild('AllowS3Invocation'));
This is basically a CDK version of the CloudFormation template laid out in this example. See the docs on the AWS SDK for the possible NotificationConfiguration parameters.
since June 2021 there is a nicer way to solve this problem. Since approx. Version 1.110.0 of the CDK it is possible to use the S3 notifications with Typescript Code:
Example:
const s3Bucket = s3.Bucket.fromBucketName(this, 'bucketId', 'bucketName');
s3Bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(lambdaFunction), {
prefix: 'example/file.txt'
});
CDK Documentation:
https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-notifications-readme.html
Pull Request:
https://github.com/aws/aws-cdk/pull/15158
Sorry I can't comment on the excellent James Irwin's answer above due to a low reputation, but I took and made it into a Construct.
The comment about "Access Denied" took me some time to figure out too, but the crux of it is that the function is S3:putBucketNotificationConfiguration, but the IAM Policy action to allow is S3:PutBucketNotification.
Here's the [code for the construct]:(https://gist.github.com/archisgore/0f098ae1d7d19fddc13d2f5a68f606ab)
import * as cr from '#aws-cdk/custom-resources';
import * as logs from '#aws-cdk/aws-logs';
import * as s3 from '#aws-cdk/aws-s3';
import * as sqs from '#aws-cdk/aws-sqs';
import * as iam from '#aws-cdk/aws-iam';
import {Construct} from '#aws-cdk/core';
// You can drop this construct anywhere, and in your stack, invoke it like this:
// const s3ToSQSNotification = new S3NotificationToSQSCustomResource(this, 's3ToSQSNotification', existingBucket, queue);
export class S3NotificationToSQSCustomResource extends Construct {
constructor(scope: Construct, id: string, bucket: s3.IBucket, queue: sqs.Queue) {
super(scope, id);
// https://stackoverflow.com/questions/58087772/aws-cdk-how-to-add-an-event-notification-to-an-existing-s3-bucket
const notificationResource = new cr.AwsCustomResource(scope, id+"CustomResource", {
onCreate: {
service: 'S3',
action: 'putBucketNotificationConfiguration',
parameters: {
// This bucket must be in the same region you are deploying to
Bucket: bucket.bucketName,
NotificationConfiguration: {
QueueConfigurations: [
{
Events: ['s3:ObjectCreated:*'],
QueueArn: queue.queueArn,
}
]
}
},
physicalResourceId: <cr.PhysicalResourceId>(id + Date.now().toString()),
},
onDelete: {
service: 'S3',
action: 'putBucketNotificationConfiguration',
parameters: {
// This bucket must be in the same region you are deploying to
Bucket: bucket.bucketName,
// deleting a notification configuration involves setting it to empty.
NotificationConfiguration: {
}
},
physicalResourceId: <cr.PhysicalResourceId>(id + Date.now().toString()),
},
policy: cr.AwsCustomResourcePolicy.fromStatements([new iam.PolicyStatement({
// The actual function is PutBucketNotificationConfiguration.
// The "Action" for IAM policies is PutBucketNotification.
// https://docs.aws.amazon.com/AmazonS3/latest/dev/list_amazons3.html#amazons3-actions-as-permissions
actions: ["S3:PutBucketNotification"],
// allow this custom resource to modify this bucket
resources: [bucket.bucketArn],
})]),
logRetention: logs.RetentionDays.ONE_DAY,
});
// allow S3 to send notifications to our queue
// https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#grant-destinations-permissions-to-s3
queue.addToResourcePolicy(new iam.PolicyStatement({
principals: [new iam.ServicePrincipal("s3.amazonaws.com")],
actions: ["SQS:SendMessage"],
resources: [queue.queueArn],
conditions: {
ArnEquals: {"aws:SourceArn": bucket.bucketArn}
}
}));
// don't create the notification custom-resource until after both the bucket and queue
// are fully created and policies applied.
notificationResource.node.addDependency(bucket);
notificationResource.node.addDependency(queue);
}
}
UPDATED: Source code from original answer will overwrite existing notification list for bucket which will make it impossible adding new lambda triggers. Here's the solution which uses event sources to handle mentioned problem.
import aws_cdk {
aws_s3 as s3,
aws_cdk.aws_lambda as lambda_
aws_lambda_event_sources as event_src
}
import path as path
class S3LambdaTrigger(core.Stack):
def __init__(self, scope: core.Construct, id: str):
super().__init__(scope, id)
bucket = s3.Bucket(
self, "S3Bucket",
block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
bucket_name='BucketName',
encryption=s3.BucketEncryption.S3_MANAGED,
versioned=True
)
fn = lambda_.Function(
self, "LambdaFunction",
runtime=lambda_.Runtime.NODEJS_10_X,
handler="index.handler",
code=lambda_.Code.from_asset(path.join(__dirname, "lambda-handler"))
)
fn.add_permission(
's3-service-principal',
principal=aws_iam.ServicePrincipal('s3.amazonaws.com')
)
fn.add_event_source(
event_src.S3EventSource(
bucket,
events=[s3.EventType.OBJECT_CREATED, s3.EventType.OBJECT_REMOVED],
filters=[s3.NotificationKeyFilter(prefix="subdir/", suffix=".txt")]
)
)
ORIGINAL:
I took ubi's solution in TypeScript and successfully translated it to Python. His solution worked for me.
#!/usr/bin/env python
from typing import List
from aws_cdk import (
core,
custom_resources as cr,
aws_lambda as lambda_,
aws_s3 as s3,
aws_iam as iam,
)
class S3NotificationLambdaProps:
def __init__(self, bucket: s3.Bucket, function: lambda_.Function, events: List[str], prefix: str):
self.bucket = bucket
self.function = function
self.events = events
self.prefix = prefix
class S3NotificationLambda(core.Construct):
def __init__(self, scope: core.Construct, id: str, props: S3NotificationLambdaProps):
super().__init__(scope, id)
self.notificationResource = cr.AwsCustomResource(
self, f'CustomResource{id}',
on_create=cr.AwsSdkCall(
service="S3",
action="S3:putBucketNotificationConfiguration",
# Always update physical ID so function gets executed
physical_resource_id=cr.PhysicalResourceId.of(f'S3NotifCustomResource{id}'),
parameters={
"Bucket": props.bucket.bucket_name,
"NotificationConfiguration": {
"LambdaFunctionConfigurations": [{
"Events": props.events,
"LambdaFunctionArn": props.function.function_arn,
"Filter": {
"Key": {"FilterRules": [{"Name": "prefix", "Value": props.prefix}]}
}}
]
}
}
),
on_delete=cr.AwsSdkCall(
service="S3",
action="S3:putBucketNotificationConfiguration",
# Always update physical ID so function gets executed
physical_resource_id=cr.PhysicalResourceId.of(f'S3NotifCustomResource{id}'),
parameters={
"Bucket": props.bucket.bucket_name,
"NotificationConfiguration": {},
}
),
policy=cr.AwsCustomResourcePolicy.from_statements(
statements=[
iam.PolicyStatement(
actions=["S3:PutBucketNotification", "S3:GetBucketNotification"],
resources=[props.bucket.bucket_arn]
),
]
)
)
props.function.add_permission(
"AllowS3Invocation",
action="lambda:InvokeFunction",
principal=iam.ServicePrincipal("s3.amazonaws.com"),
source_arn=props.bucket.bucket_arn,
)
# don't create the notification custom-resource until after both the bucket and lambda
# are fully created and policies applied.
self.notificationResource.node.add_dependency(props.bucket)
self.notificationResource.node.add_dependency(props.function)
# Usage:
s3NotificationLambdaProps = S3NotificationLambdaProps(
bucket=bucket_,
function=lambda_fn_,
events=['s3:ObjectCreated:*'],
prefix='foo/'
)
s3NotificationLambda = S3NotificationLambda(
self, "S3NotifLambda",
self.s3NotificationLambdaProps
)
Here is a python solution for adding / replacing a lambda trigger to an existing bucket including the filter. #James Irwin your example was very helpful.
Thanks to #JørgenFrøland for pointing out that the custom resource config will replace any existing notification triggers based on the boto3 documentation https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.BucketNotification.put
One note is he access denied issue is
because if you do putBucketNotificationConfiguration action the policy creates a s3:PutBucketNotificationConfiguration action but that action doesn't exist https://github.com/aws/aws-cdk/issues/3318#issuecomment-584737465
Same issue happens if you set the policy using AwsCustomResourcePolicy.fromSdkCalls
I've added a custom policy that might need to be restricted further.
s3_bucket = s3.Bucket.from_bucket_name(
self, 's3-bucket-by-name', 'existing-bucket-name')
trigger_lambda = _lambda.Function(
self,
'{id}-s3-trigger-lambda',
environment=lambda_env,
code=_lambda.Code.from_asset('./ladle-sink/'),
runtime=_lambda.Runtime.PYTHON_3_7,
handler='lambda_function.lambda_handler',
memory_size=512,
timeout=core.Duration.minutes(3))
trigger_lambda.add_permission(
's3-trigger-lambda-s3-invoke-function',
principal=iam.ServicePrincipal('s3.amazonaws.com'),
action='lambda:InvokeFunction',
source_arn=base_resources.incoming_documents_bucket.bucket_arn)
custom_s3_resource = _custom_resources.AwsCustomResource(
self,
's3-incoming-documents-notification-resource',
policy=_custom_resources.AwsCustomResourcePolicy.from_statements([
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
resources=['*'],
actions=['s3:PutBucketNotification']
)
]),
on_create=_custom_resources.AwsSdkCall(
service="S3",
action="putBucketNotificationConfiguration",
parameters={
"Bucket": s3_bucket.bucket_name,
"NotificationConfiguration": {
"LambdaFunctionConfigurations": [
{
"Events": ['s3:ObjectCreated:*'],
"LambdaFunctionArn": trigger_lambda.function_arn,
"Filter": {
"Key": {
"FilterRules": [
{'Name': 'suffix', 'Value': 'html'}]
}
}
}
]
}
},
physical_resource_id=_custom_resources.PhysicalResourceId.of(
f's3-notification-resource-{str(uuid.uuid1())}'),
region=env.region
))
custom_s3_resource.node.add_dependency(
trigger_lambda.permissions_node.find_child(
's3-trigger-lambda-s3-invoke-function'))
Thanks to the great answers above, see below for a construct for s3 -> lambda notification. It can be used like
const fn = new SingletonFunction(this, "Function", {
...
});
const bucket = Bucket.fromBucketName(this, "Bucket", "...");
const s3notification = new S3NotificationLambda(this, "S3Notification", {
bucket: bucket,
lambda: function,
events: ['s3:ObjectCreated:*'],
prefix: "some_prefix/"
})
Construct (drop-in to your project as a .ts file)
import * as cr from "#aws-cdk/custom-resources";
import * as logs from "#aws-cdk/aws-logs";
import * as s3 from "#aws-cdk/aws-s3";
import * as sqs from "#aws-cdk/aws-sqs";
import * as iam from "#aws-cdk/aws-iam";
import { Construct } from "#aws-cdk/core";
import * as lambda from "#aws-cdk/aws-lambda";
export interface S3NotificationLambdaProps {
bucket: s3.IBucket;
lambda: lambda.IFunction;
events: string[];
prefix: string;
}
export class S3NotificationLambda extends Construct {
constructor(scope: Construct, id: string, props: S3NotificationLambdaProps) {
super(scope, id);
const notificationResource = new cr.AwsCustomResource(
scope,
id + "CustomResource",
{
onCreate: {
service: "S3",
action: "putBucketNotificationConfiguration",
parameters: {
// This bucket must be in the same region you are deploying to
Bucket: props.bucket.bucketName,
NotificationConfiguration: {
LambdaFunctionConfigurations: [
{
Events: props.events,
LambdaFunctionArn: props.lambda.functionArn,
Filter: {
Key: {
FilterRules: [{ Name: "prefix", Value: props.prefix }],
},
},
},
],
},
},
physicalResourceId: <cr.PhysicalResourceId>(
(id + Date.now().toString())
),
},
onDelete: {
service: "S3",
action: "putBucketNotificationConfiguration",
parameters: {
// This bucket must be in the same region you are deploying to
Bucket: props.bucket.bucketName,
// deleting a notification configuration involves setting it to empty.
NotificationConfiguration: {},
},
physicalResourceId: <cr.PhysicalResourceId>(
(id + Date.now().toString())
),
},
policy: cr.AwsCustomResourcePolicy.fromStatements([
new iam.PolicyStatement({
// The actual function is PutBucketNotificationConfiguration.
// The "Action" for IAM policies is PutBucketNotification.
// https://docs.aws.amazon.com/AmazonS3/latest/dev/list_amazons3.html#amazons3-actions-as-permissions
actions: ["S3:PutBucketNotification", "S3:GetBucketNotification"],
// allow this custom resource to modify this bucket
resources: [props.bucket.bucketArn],
}),
]),
}
);
props.lambda.addPermission("AllowS3Invocation", {
action: "lambda:InvokeFunction",
principal: new iam.ServicePrincipal("s3.amazonaws.com"),
sourceArn: props.bucket.bucketArn,
});
// don't create the notification custom-resource until after both the bucket and queue
// are fully created and policies applied.
notificationResource.node.addDependency(props.bucket);
notificationResource.node.addDependency(props.lambda);
}
}
based on the answer from #ubi
in case of you don't need the SingletonFunction but Function + some cleanup
call like this:
const s3NotificationLambdaProps = < S3NotificationLambdaProps > {
bucket: bucket,
lambda: lambda,
events: ['s3:ObjectCreated:*'],
prefix: '', // or put some prefix
};
const s3NotificationLambda = new S3NotificationLambda(this, `${envNameUpperCase}S3ToLambdaNotification`, s3NotificationLambdaProps);
and the construct will be like this:
import * as cr from "#aws-cdk/custom-resources";
import * as s3 from "#aws-cdk/aws-s3";
import * as iam from "#aws-cdk/aws-iam";
import { Construct } from "#aws-cdk/core";
import * as lambda from "#aws-cdk/aws-lambda";
export interface S3NotificationLambdaProps {
bucket: s3.IBucket;
lambda: lambda.Function;
events: string[];
prefix: string;
}
export class S3NotificationLambda extends Construct {
constructor(scope: Construct, id: string, props: S3NotificationLambdaProps) {
super(scope, id);
const notificationResource = new cr.AwsCustomResource(
scope,
id + "CustomResource", {
onCreate: {
service: "S3",
action: "putBucketNotificationConfiguration",
parameters: {
// This bucket must be in the same region you are deploying to
Bucket: props.bucket.bucketName,
NotificationConfiguration: {
LambdaFunctionConfigurations: [{
Events: props.events,
LambdaFunctionArn: props.lambda.functionArn,
Filter: {
Key: {
FilterRules: [{
Name: "prefix",
Value: props.prefix
}],
},
},
}, ],
},
},
physicalResourceId: < cr.PhysicalResourceId > (
(id + Date.now().toString())
),
},
onDelete: {
service: "S3",
action: "putBucketNotificationConfiguration",
parameters: {
// This bucket must be in the same region you are deploying to
Bucket: props.bucket.bucketName,
// deleting a notification configuration involves setting it to empty.
NotificationConfiguration: {},
},
physicalResourceId: < cr.PhysicalResourceId > (
(id + Date.now().toString())
),
},
policy: cr.AwsCustomResourcePolicy.fromStatements([
new iam.PolicyStatement({
// The actual function is PutBucketNotificationConfiguration.
// The "Action" for IAM policies is PutBucketNotification.
// https://docs.aws.amazon.com/AmazonS3/latest/dev/list_amazons3.html#amazons3-actions-as-permissions
actions: ["S3:PutBucketNotification", "S3:GetBucketNotification"],
// allow this custom resource to modify this bucket
resources: [props.bucket.bucketArn],
}),
]),
}
);
props.lambda.addPermission("AllowS3Invocation", {
action: "lambda:InvokeFunction",
principal: new iam.ServicePrincipal("s3.amazonaws.com"),
sourceArn: props.bucket.bucketArn,
});
// don't create the notification custom-resource until after both the bucket and lambda
// are fully created and policies applied.
notificationResource.node.addDependency(props.bucket);
notificationResource.node.addDependency(props.lambda);
}
}
With the newer functionality, in python this can now be done as:
bucket = aws_s3.Bucket.from_bucket_name(
self, "bucket", "bucket-name"
)
bucket.add_event_notification(
aws_s3.EventType.OBJECT_CREATED,
aws_s3_notifications.LambdaDestination(your_lambda),
aws_s3.NotificationKeyFilter(
prefix="prefix/path/",
),
)
At the time of writing, the AWS documentation seems to have the prefix arguments incorrect in their examples so this was moderately confusing to figure out.
Thanks to #Kilian Pfeifer for starting me down the right path with the typescript example.
I used CloudTrail for resolving the issue, code looks like below and its more abstract:
const trail = new cloudtrail.Trail(this, 'MyAmazingCloudTrail');
const options: AddEventSelectorOptions = {
readWriteType: cloudtrail.ReadWriteType.WRITE_ONLY
};
// Adds an event selector to the bucket
trail.addS3EventSelector([{
bucket: bucket, // 'Bucket' is of type s3.IBucket,
}], options);
bucket.onCloudTrailWriteObject('MyAmazingCloudTrail', {
target: new targets.LambdaFunction(functionReference)
});
This is CDK solution.
Get a grab of existing bucket using fromBucketAttributes
Then for your bucket, use addEventNotification to trigger your lambda.
declare const myLambda: lambda.Function;
const bucket = s3.Bucket.fromBucketAttributes(this, 'ImportedBucket', {
bucketArn: 'arn:aws:s3:::my-bucket',
});
// now you can just call methods on the bucket
bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(myLambda), {prefix: 'home/myusername/*'});
More details can be found here
AWS now supports s3 eventbridge events, which allows for adding a source s3 bucket by name. So this worked for me. Note that you need to enable eventbridge events manually for the triggering s3 bucket.
new Rule(this, 's3rule', {
eventPattern: {
source: ['aws.s3'],
detail: {
'bucket': {'name': ['existing-bucket']},
'object': {'key' : [{'prefix' : 'prefix'}]}
},
detailType: ['Object Created']
},
targets: [new targets.LambdaFunction(MyFunction)]
}
);