I currently have a dynamodb table that's been in use for a couple of years, originally created in the console. It contains lots of valuable data. It uses a stream to periodically send a snapshot using a lambda trigger of the table to s3 for analytics. The table itself is heavily used by end users to access their data.
I want to migrate my solution into CDK. The options I want to explore:
When you use the Table.fromTableArn construct, you don't get access to the table stream arn so it's impossible to attach a lambda trigger. Is there a way around this?
Is there a way to clone the dynamoDB table contents in my CDK stack so that my copy will start off in exactly the same state as the original? Then I can add and manage the stream etc in CDK no problem.
It's worth checking my assumption that these are the only 2 options.
Subscribing Lambda to an existing Dynamo Table:
We don't need to have table created within same stack. We can't use addEventSource on lambda but we can use addEventSourceMapping and add necessary policies to Lambda, which is what addEventSource does behind the scenes.
const streamsArn =
"arn:aws:dynamodb:us-east-1:110011001100:table/test/stream/2021-03-18T06:25:21.904";
const myLambda = new lambda.Function(this, "my-lambda", {
code: new lambda.InlineCode(`
exports.handler = (event, context, callback) => {
console.log('event',event)
callback(null,'10')
}
`),
handler: "index.handler",
runtime: lambda.Runtime.NODEJS_10_X,
});
const eventSoruce = myLambda.addEventSourceMapping("test", {
eventSourceArn: streamsArn,
batchSize: 5,
startingPosition: StartingPosition.TRIM_HORIZON,
bisectBatchOnError: true,
retryAttempts: 10,
});
const roleUpdates = myLambda.addToRolePolicy(
new iam.PolicyStatement({
actions: [
"dynamodb:DescribeStream",
"dynamodb:GetRecords",
"dynamodb:GetShardIterator",
"dynamodb:ListStreams",
],
resources: [streamsArn],
})
);
Importing existing DynamoDb into CDK:
We re-write dynamo db with same attributes in cdk, synth to generate Cloudformation and use resource import to import an existing resources into a stack. Here is an SO answer
Related
How I can create an Athena data source in AWS CDK which is a JDBC connection to a MySQL database using the AthenaJdbcConnector?
I believe I can use aws-sam's CfnApplication to create the AthenaJdbcConnector Lambda, but how can I connect it to Athena?
I notice a lot of Glue support in CDK which would transfer to Athena (data catalog), and there are several CfnDataSource types in other modules such as QuickSight, but I'm not seeing anything under Athena in CDK.
See the image and references below.
References:
https://docs.aws.amazon.com/athena/latest/ug/athena-prebuilt-data-connectors-jdbc.html
https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-jdbc
https://serverlessrepo.aws.amazon.com/applications/us-east-1/292517598671/AthenaJdbcConnector
I have been playing with the same issue. Here is what I did to create the Lambda for federated queries (Typescript):
const vpc = ec2.Vpc.fromLookup(this, 'my-project-vpc', {
vpcId: props.vpcId
});
const cluster = new rds.ServerlessCluster(this, 'AuroraCluster', {
engine: rds.DatabaseClusterEngine.AURORA_POSTGRESQL,
parameterGroup: rds.ParameterGroup.fromParameterGroupName(this, 'ParameterGroup', 'default.aurora-postgresql10'),
defaultDatabaseName: 'MyDB',
vpc,
vpcSubnets: {
onePerAz: true
},
scaling: {autoPause: cdk.Duration.seconds(0)} // Optional. If not set, then instance will pause after 5 minutes
});
let password = cluster.secret!.secretValueFromJson('password').toString()
let spillBucket = new Bucket(this, "AthenaFederatedSpill")
let lambdaApp = new CfnApplication(this, "MyDB", {
location: {
applicationId: "arn:aws:serverlessrepo:us-east-1:292517598671:applications/AthenaJdbcConnector",
semanticVersion: "2021.42.1"
},
parameters: {
DefaultConnectionString: `postgres://jdbc:postgresql://${cluster.clusterEndpoint.hostname}/MyDB?user=postgres&password=${password}`,
LambdaFunctionName: "crossref_federation",
SecretNamePrefix: `${cluster.secret?.secretName}`,
SecurityGroupIds: `${cluster.connections.securityGroups.map(value => value.securityGroupId).join(",")}`,
SpillBucket: spillBucket.bucketName,
SubnetIds: vpc.privateSubnets[0].subnetId
}
})
This creates the lambda with a default connection string like you would have it, if you used the AWS Console wizard in Athena to connect to a DataSource. Unfortunately it is NOT possible to add a Athena-catalog specific connection string via CDK. It should be set as an Environment Variable on the Lambda, and I found no way to do that. The Application template simply don't allow it, so this is a post-process by hand. I would sure like to hear from anybody if they have a solution for that!
Also notice that I add the user/password in the jdbc URL directly. I wanted to use SecretsManager, but because the Lambda is deployed in a VPC, it simply refuses to connect to the secretsmanager. I think this might be solvable by added a private VPN connection to SSM. Again - I would like to hear from anybody have tried that.
I'm migrating my cloud solution to cdk. I can see how to add a stream to a new DynamoDB in the constructor through the TableProps:
const newTable = new dynamodb.Table(this, 'new Table', {
tableName: 'streaming',
partitionKey: { name : 'id', type: dynamodb.AttributeType.NUMBER },
stream: StreamViewType.NEW_AND_OLD_IMAGES,
})
but there is no apparent way to enable a stream on an existing DynamoDB. I can't seem to access the TableProps on an existing item.
const sandpitTable = dynamodb.Table.fromTableArn(this, 'sandpitTable', 'arn:aws:dynamodb:ap-southeast-2:xxxxxxxxxxxxxx:table/Sandpit');
sandpitTable.grantStreamRead(streamLambda);
// sandpitTable. ??? what to do?
How can this be achieved? And how does the solution take into account disaster recovery and prevent accidental deletion of the Dynamo DB that is not possible when using the console.
Enabling streams is just another attribute of resource 'AWS::DynamoDB::Table' in CloudFormation and I don't believe we can make changes to a resource that is created in a stack (or manually) from another cloudformation/cdk stack unless we import the resource.
Here is documentation. I can try and summarize.
Assume we have an existing cdk project which is deployed without Metadata resource cdk --no-version-reporting deploy
Assuming we have Dynamo table 'streaming' with partiion key 'id' as you stated.
Adding below cdk code with same attributes of original table like RCU, WCU, keys, etc. For simplicity I just gave name and key and removalPolicy is must
const myTable = new dynamodb.Table(this, "dynamo-table", {
tableName: "streaming",
partitionKey: { name: "id", type: dynamodb.AttributeType.NUMBER },
removalPolicy: cdk.RemovalPolicy.RETAIN,
});
We can now synth and generate the CloudFormation by default into cdk.out folder cdk --no-version-reporting synth
Grab the logical Id from .json file in my case it is dynamotableF6720B98
Create ChangeSet set with right table name and logical id
aws cloudformation create-change-set --stack-name HelloCdkStack --change-set-name ImportChangeSet --change-set-type IMPORT --resources-to-import "[{\"ResourceType\":\"AWS::DynamoDB::Table\",\"LogicalResourceId\":\"dynamotableF6720B98\",\"ResourceIdentifier\":{\"TableName\":\"streaming\"}}]" --template-body file://cdk.out/HelloCdkStack.template.json
Execute change set
aws cloudformation execute-change-set --change-set-name ImportChangeSet --stack-name HelloCdkStack
Best to check the drift and make necessary chagnes to
aws cloudformation detect-stack-drift --stack-name HelloCdkStack
To your other question of preventing accidental deletion, we can simply add deletion policy to avoid dynamo table getting deleted when stack/resource is deleted.
removalPolicy: RemovalPolicy.RETAIN
I'm trying to use AWS CDK to create a new lambda tied to already existing AWS resources which were not created using CDK and that are part of a different stack.
Can I trigger my lambda from an already existing user pool using CDK? I've imported the user pool to my new stack using:
const userPool = UserPool.fromUserPoolArn(this, 'poolName, 'arn:aws:cognito-idp:eu-west-1:1234567890:userpool/poolName')
However, this gives me an IUserPool which does not have the addTrigger method. Is there a way to convert this into a UserPool in order to be able to trigger the lambda (since I can see that UserPool has the addTrigger method)?
I have seen that it is possible to e.g. grant permissions for my new lambda to read/write into an existing DynamoDB table using CDK. And I don't really understand the difference here: DynamoDB is an existing AWS resource and I'm importing it to the new stack using CDK and then allowing my new lambda to modify it. The Cognito User Pool is also an existing AWS resource, and I am able to import it in CDK but it seems that I'm not able to modify it? Why?
This was discussed in this issue. You can add triggers to an existing User Pool using a Custom Resource:
import * as CustomResources from '#aws-cdk/custom-resources';
import * as Cognito from '#aws-cdk/aws-cognito';
import * as Iam from '#aws-cdk/aws-iam';
const userPool = Cognito.UserPool.fromUserPoolId(this, "UserPool", userPoolId);
new CustomResources.AwsCustomResource(this, "UpdateUserPool", {
resourceType: "Custom::UpdateUserPool",
onCreate: {
region: this.region,
service: "CognitoIdentityServiceProvider",
action: "updateUserPool",
parameters: {
UserPoolId: userPool.userPoolId,
LambdaConfig: {
PreSignUp: preSignUpHandler.functionArn
},
},
physicalResourceId: CustomResources.PhysicalResourceId.of(userPool.userPoolId),
},
policy: CustomResources.AwsCustomResourcePolicy.fromSdkCalls({ resources: CustomResources.AwsCustomResourcePolicy.ANY_RESOURCE }),
});
const invokeCognitoTriggerPermission = {
principal: new Iam.ServicePrincipal('cognito-idp.amazonaws.com'),
sourceArn: userPool.userPoolArn
}
preSignUpHandler.addPermission('InvokePreSignUpHandlerPermission', invokeCognitoTriggerPermission)
You can also modify other User Pool settings with this method.
I'm using the Serverless framework to deploy my functions on the AWS Lambda
I'm trying to create a trigger automatically for each version of my Lambda functions published.
When I deploy my serverless app, the Lambda function and the triggers are created (in this case my AWS IOT trigger), as we can see on the following image:
But for my published version of the lambda function the trigger doesn't exist, only the resources:
I don't want to create new triggers every time I publish a new lambda version.
So, there is any way to create the triggers for my versioned lambdas too? And if is possible, disable the old ones using the Serverless framework?
my serverless.yml file:
service: serverless-lambdas
provider:
name: aws
runtime: nodejs6.10
iamRoleStatements:
- Effect: "Allow"
Action:
- "ses:*"
- "iot:*"
Resource:
- "*"
functions:
function1:
name: "function1"
handler: function1/handler.function1
events:
- iot:
name: "iotEvent1"
sql: "SELECT EXAMPLE"
sqlVersion: "2016-03-23"
enabled: true
UPDATE
I got a similar problem when I was trying to create triggers programmatically using my own AWS Lambda.
I got stuck on this when I saw that the problem was with my trigger that which had no permission to invoke the published Lambda function. So I needed to add the permission for the trigger first with the method add-permission. (This is not clearly written on the AWS docs :/).
So, before I added the the trigger on the Lambda, I used the following method (in node.js):
const addPermission = (ruleName) => {
const thingArn = `arn:aws:iot:${IOT_REGION}:${SOURCE_ACCOUNT}:rule/${ruleName}`;
const params = {
Action: "lambda:InvokeFunction",
FunctionName: LAMBDA_NAME,
Principal: "iot.amazonaws.com",
SourceAccount: SOURCE_ACCOUNT,
SourceArn: thingArn,
StatementId: `iot-sd-${Math.random().toString(36).substring(2) + Date.now().toString(36)}`
};
return new Promise((resolve, reject) => {
lambda.addPermission(params).promise().then(result => {
resolve(result)
}).catch(err => reject(err))
});
};
I tested the same function for the Serverless framework, and Shazam! my triggers were published! We can do something like this for now while the Serverless code is not updated.
In this way, this problem will need to be solved on the Serverless source-code and I will try to do it ASAP.
From what I checked this is the default behavior for the AWS Lambda functions, so there is no issue with the Serverless framework.
Every time I publish a Lambda function, there is no way to create the trigger events automatically.
For further information, we can read the documentation of Versioning aliases.
Can someone tell me how to get access to AWS credentials in an AWS Lambda function?
I've searched the internet thoroughly but still haven't found anything that has helped me.
Im writing the function in Java. I think I should have access to the credentials with the context object in the HandleRequest method.
If it helps, I want to invoke a DynamoDB client and upload a record to the database.
I came into the same problem myself recently.
This certainly is a blind spot in AWS's Lambda documentation for Java, in my opinion.
This snippet in Java should work for you, assuming you're using the AWS SDK for Java Document API :
DynamoDB dynamodb = new DynamoDB(
new AmazonDynamoDBClient(new EnvironmentVariableCredentialsProvider()));
The main takeaway is to use the EnvironmentVariableCredentialsProvider to access the required credentials to access your other AWS resources within the AWS Lambda container. The Lambda containers are shipped with credentials as environment variables, and this is sufficient in retrieving them.
Note: This creates a DynamoDB instance that only sees resources in the default region. To create one for a specific region, use this (assuming you want to access DynamoDB's in the ap-northeast-1 region):
DynamoDB dynamodb = new DynamoDB(
Regions.getRegion(Regions.AP_NORTHEAST_1).createClient(
AmazonDynamoDBClient.class,
new EnvironmentVariableCredentialsProvider(),
new ClientConfiguration()));
Your Lambda function's permissions are controlled by the IAM Role it executes as. Either add Dynamo PutItem permission to the current role, or create a new role for this purpose.
After giving permissions to the Role, you don't need to write special code to handle credentials, just use the AWS SDK normally. For example:
var AWS = require("aws-sdk");
exports.handler = function(event, context) {
var dynamodb = new AWS.DynamoDB();
var putItemParams = {
"TableName": "Scratch",
"Item": {
"Id": {
"S": "foo"
},
"Text": {
"S": "bar"
}
}
};
dynamodb.putItem(putItemParams, function (err, response) {
if (err) {
context.fail("dynamodb.putItem failed: " + err);
} else {
context.succeed("dynamodb.putItem succeeded");
}
});
};
Is sufficient to put an item in a DynamoDB table, with the correct Role permissions.
Adding to #Gordon Tai's answer, using the current api using AmazonDynamoDBClientBuilder this looks like:
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
.withCredentials(new EnvironmentVariableCredentialsProvider())
.withRegion(Regions.US_EAST_1)
.build();
DynamoDB dynamoDB = new DynamoDB(client);