I need a component for creating a large number of CodeCommit users. CloudFormation doesn't support adding a public SSH key for an IAM user, so I have to create my own. CDK comes with AwsCustomResource, which does the heavy lifting of creating the Lambda that handles the required CloudFormation events. In other words, my code would be something like:
import { User } from 'aws-cdk-lib/aws-iam';
import { AwsCustomResource } from 'aws-cdk-lib/custom-resources';
import { Construct } from 'constructs';
export interface CodeCommitUserProps {
userName: string;
emailAddress: string;
sshPublicKey: string;
}
export class CodeCommitUser extends Construct {
constructor(scope: Construct, id: string, props: CodeCommitUserProps) {
super(scope, id);
const user = new User(this, 'codecommit-' + props.userName, {
userName: 'codecommit-' + props.userName,
path: '/git/users'
});
}
const custom = new AwsCustomResource(this, ... );
}
Now if I call new CodeCommitUser(...) a few hundred times, I would assume that there will be one CloudFormation event Lambda per user, even if all of them are identical. Is there a way to reuse the Lambdas created by AwsCustomResource if I need multiple copies of the custom resource?
I would assume that there will be one CloudFormation event Lambda per user, even if all of them are identical.
Actually, no. CDK creates a single lambda function, no matter how many times CodeCommitUser is instantiated. How does CDK manage this? Under the hood, CDK uses a SingletonFunction for the AWSCustomResource provider (see the github source). Singleton Functions are guaranteed to be added to the stack "once and only once, irrespective of how many times the construct is declared to be part of the stack"
Is there a way to reuse the Lambdas created by AwsCustomResource if I need multiple copies of the custom resource?
Again, reuse happens automagically. You can prove this to yourself by cdk synth-ing the stack with multiple CodeCommitUsers defined. Then look in the cdk.out directory for the outputted CloudFormation template. The template have only one AWS::Lambda::Function resource defined (assuming your app doesn't use lambdas elsewhere).
You can create your custom resource lambda and deploy it in a separate template. Then you can call on it from what ever template you want.
You can call this resource as many times you want from a single template.
You can either send a list of users in one go, or create a resource for each user (probably not ideal if you talking hundreds).
Related
I am creating two Stack using AWS CDK. I use the first Stack to create an S3 bucket and upload lambda Zip file to the bucket using BucketDeployment construct, like this.
//FirstStack
const deployments = new BucketDeployment(this, 'LambdaDeployments', {
destinationBucket: bucket,
destinationKeyPrefix: '',
sources: [
Source.asset(path)
],
retainOnDelete: true,
extract: false,
accessControl: BucketAccessControl.PUBLIC_READ,
});
I use the second Stack just to generate CloudFormation template to my clients. In the second Stack, I want to create a Lambda function with parameters S3 bucket name and key name of the Lambda zip I uploaded in the 1st stack.
//SecondStack
const lambdaS3Bucket = "??"; //TODO
const lambdaS3Key = "??"; //TODO
const bucket = Bucket.fromBucketName(this, "Bucket", lambdaS3Bucket);
const lambda = new Function(this, "LambdaFunction", {
handler: 'index.handler',
runtime: Runtime.NODEJS_16_X,
code: Code.fromBucket(
bucket,
lambdaS3Key
),
});
How do I refer the parameters automatically from 2nd Lambda?
In addition to that, the lambdaS3Bucket need to have AWS::Region parameters so that my clients can deploy it in any region (I just need to run the first Stack in the region they require).
How do I do that?
I had a similar usecase to this one.
The very simple answer is to hardcode the values. The bucketName is obvious.
The lambdaS3Key You can look up in the synthesized template of the first stack.
More complex answer is to use pipelines for this. I've did this and in the build step of the pipeline I extracted all lambdaS3Keys and exported them as environment variable, so in the second stack I could reuse these in the code, like:
code: Code.fromBucket(
bucket,
process.env.MY_LAMBDA_KEY
),
I see You are aware of this PR, because You are using the extract flag.
Knowing that You can probably reuse this property for Lambda Key.
The problem of sharing the names between the stacks in different accounts remains nevertheless. My suggestion is to use pipelines and the exported constans there in the different steps, but also a local build script would do the job.
Do not forget to update the BucketPolicy and KeyPolicy if You use encryption, otherwise the customer account won't have the access to the file.
You could also read about the AWS Service Catalog. Probably this would be a esier way to share Your CDK products to Your customers (CDK team is going to support the out of the box lambda sharing next on)
I'm exploring how refactorable CDK applications are. Suppose I defined a custom construct (a stack) to create an EKS cluster. Let's call it EksStack. Ideally, I'd create the role to be associated with the cluster and the EKS cluster itself, as described by the following snippet (I'm using Scala instead of Java, so the snippets are going to be in Scala syntax):
class EksStack (scope: Construct, id: String, props: StackProps) extends Stack(scope, id, props) {
private val role = new Role(this, "eks-role", RoleProps.builder()
.description(...)
.managedPolicies(...)
.assumedBy(...)
.build()
)
private val cluster = new Cluster(this, "eks-cluster", ClusterProps.builder()
.version(...)
.role(role)
.defaultCapacityType(DefaultCapacityType.EC2)
.build()
)
}
When I synthetize the application, I can see that the generated template contains the definition of the VPC, together with the Elastic IPs, NATs, Internet Gateways, and so on.
Now suppose that I want to refactor EksStack and have a different stack, say VpcStack, explicitly create the VPC:
class VpcStack (scope: Construct, id: String, props: StackProps) extends Stack(scope, id, props) {
val vpc = new Vpc(this, VpcId, VpcProps.builder()
.cidr(...)
.enableDnsSupport(true)
.enableDnsHostnames(true)
.maxAzs(...)
.build()
)
}
Ideally, the cluster in EksStack would just be using the reference to the VPC created by VpcStack, something like (note the new call to vpc() in the builder of cluster):
class EksStack (scope: Construct, id: String, props: StackProps, vpc: IVpc) extends Stack(scope, id, props) {
private val role = new Role(this, "eks-role", RoleProps.builder()
.description(...)
.managedPolicies(...)
.assumedBy(...)
.build()
)
private val cluster = new Cluster(this, "eks-cluster", ClusterProps.builder()
.version(...)
.role(role)
.vpc(vpc)
.defaultCapacityType(DefaultCapacityType.EC2)
.build()
)
}
This obviously doesn't work, as CloudFormation would delete the VPC created by EksStack in favor of the one created by VpcStack. I read here and there and tried to add a retain policy in EksStack and to override the logical ID of the VPC in VpcStack, using the ID I originally saw in the CloudFormation template for EksStack:
val cfnVpc = cluster.getVpc.getNode.getDefaultChild.asInstanceOf[CfnVPC]
cfnVpc.applyRemovalPolicy(RemovalPolicy.RETAIN)
and
val cfnVpc = vpc.getNode.getDefaultChild.asInstanceOf[CfnVPC]
cfnVpc.overrideLogicalId("LogicalID")
and then retried the diff. Again, it seems that the VPC is deleted and re-created.
Now, I saw that it is possible to migrate CloudFormation resources (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/refactor-stacks.html) using the "Import resources into stack" action. My question is: can I move the creation of a resource from a stack to another in CDK without re-creating it?
EDIT:
To elaborate a bit on my problem, when I define the VPC in VpcStack, I'd like CDK to think that the resource was created by VpcStack instead ok EksStack. Something like moving the definition of it from one stack to another without having CloudFormation delete the original one to re-create it. In my use case, I'd have a stack define a create initially (either explicitly or implicitly, such as my VPC), but then, after I while, I might want to refactor my application, moving the creation of that resource in a dedicated stack. I'm trying to understand if this moving always leads to the resource being re-created of if there's any way to avoid it.
I think when it comes to refactorability when CDK and CloudFormation in general, especially multi stack configurations, there are a few principles to keep in mind.
The entire app should be able to be completely deleted and recreated. All data management is handled in the app, there is no manual processes that need to occur.
Don't always rely on auto interstack dependency management using Stack exports. I like to classify CloudFormation dependencies into two categories: Hard and soft. Hard dependencies means that you cannot delete the resource because the things using it will prevent it from happening. Soft dependencies are the opposite, the resource could be deleted and recreated without issue even though something else is using it. Hard dependency examples: VPC, Subnets. Soft dependency examples: Topic/Queue/Role.
You'll have a better time passing stack soft dependencies as Stack parameters of SSM Parameter type because you'll be able to update the stack providing the dependencies independent of those using it. Whereas you get into a deadlock when using default stack export method. You can't delete the resources because something else is importing it. So you end up having to do annoying things to make it work like deploying once with it duplicated then deploying again deleting the old stuff. It requires a little extra work to use SSM Parameters without causing stack exports but it is worth it long term for soft dependencies.
For hard dependencies, I disagree with using a lookup because you really do want to prevent deletion if something is using it bc you'll end up with a DELETE_FAILED stack and that is a terrible place to end up. So for things like VPC/Subnets, I think it's really important to actually use stack export/import technique and if you do need to recreate your VPC because of a change, if you followed principle 1, you just need to do a CDK destroy then deploy and all will be good because you built your CDK app to be fully recreatable.
When it comes to recreatability with data, CustomResources are your friend.
I'm not sure if I understand the problem, but if you are trying to reference an existing resource you can use a context query. (e.g. Vpc.fromLookup).
https://docs.aws.amazon.com/cdk/latest/guide/context.html
Additionally, if you would like to use the Vpc created from VpcStack inside of EksStack you can output the vpc id from the VpcStack and use the context query in the eks stack that way.
this is C# code but the principal is the same.
var myVpc = new Vpc(...);
new CfnOutput(this, "MyVpcIdOutput", new CfnOutputProps()
{
ExportName = "VpcIdOutput",
Value = myVpc.VpcId
}
and then when you create the EksStack you can import the vpc id that you previously exported.
new EksStack(this, "MyCoolStack", new EksStackProps()
{
MyVpcId = Fn.ImportValue("VpcIdOutput")
}
where EksStackProps is
public class EksStackProps
{
public string MyVpcId { get; set; }
}
Context
I have created a AWS Logs SubscriptionFilter using CDK. I am now trying to create a metric/alarm for some of the metrics for this resource.
Problem
All the metrics I am interested in (see ForwardedLogEvents, DeliveryErrors, DeliveryThrottling in the Monitoring AWS Logs with CloudWatch Metrics docs) requires these dimensions to be specified:
LogGroupName
DestinationType
FilterName
The first two are easy to specify since the LogGroupName is also required while creating the construct and DestinationType in my case is just Lambda. However, I see no way to get FilterName using CDK.
Using CloudWatch, I see that the FilterName is like MyStackName-MyLogicalID29669D87-GCMA0Q4KKALH. So I can't directly specify it using a Fn.ref (since I don't know the logical id). Using CloudFormation, I could have directly done Ref: LogicalId.
I also don't see any properties on the SubscriptionFilter object that will return this (unlike most other CDK constructs this one seems pretty bare and returns absolutely no information about the resource).
There are also no metric* methods on SubscriptionFilter object (unlike other standard constructs like Lambda functions, S3 buckets etc.), so I have to manually specify the Metric object. See for example: CDK metric objects docs.
The CDK construct (and the underlying CloudFormation resource: AWS::Logs::SubscriptionFilter) does not let me specify the FilterName - so I can't use a variable to specify it also and the name is dynamically generated.
Example code that is very close to what I need:
const metric = new Metric({
namespace: 'AWS/Logs',
metricName: 'ForwardedLogEvents',
dimensions: {
DestinationType: 'Lambda',
// I know this value since I specified it while creating the SubscriptionFilter
LogGroupName: 'MyLogGroupName',
FilterName: Fn.ref('logical-id-wont-work-since-it-is-dynamic-in-CDK')
}
})
Question
How can I figure out how to acquire the FilterName property to construct the Metric object?
Or otherwise, is there another way to go about this?
I was able to work around this by using Stack#getLogicalId method.
Example code
In Kotlin, as an extension function for any Construct):
fun Construct.getLogicalId() = Stack.of(this).getLogicalId(this.node.defaultChild as CfnElement)
... and then use it with any Construct:
val metric = Metric.Builder.create()
.namespace("AWS/Logs")
.metricName("ForwardedLogEvents")
.dimensions(mapOf(
"DestinationType" to "Lambda",
"LogGroupName" to myLogGroup.logGroupName,
"FilterName" to mySubscriptionFilter.getLogicalId()
))
.statistic("sum")
.build()
When using CloudFormation templates, I find the "Custom Resource" feature, with its Lambda backing function implementation, very useful to handle all kinds of tasks that CloudFormation does not provide good support for.
Usually, I use custom resources to setup things during stack creation (such as looking up AMI names) or clean up things during deletion (such as removing objects from S3 or Route53 that would block deletion) - and this works great.
But when I try to actually use a "custom resource" to manage an actual custom resource, that has to be created during stack creation, deleted during stack deletion, and - this is where the problem lies - sometimes updated with new values during a stack update, the CloudFormation integration behaves unexpectedly and causes the custom resource to fail.
The problem seems to be that during a stack update where one of the custom resource properties has changed, during the stack's UPDATE_IN_PROGRESS stage, CloudFormation sends an update event to the backing Lambda function, with all values set correctly and a copy of the old values sent as well. But after the update completes, CloudFormation starts the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS stage and sends the backing Lambda function a delete event (RequestType set to Delete).
When that happens, the backing lambda function assumes the stack is being deleted and removes the custom resource. The result is that after an update the custom resource is gone.
I've looked at the request data in the logs, and the "cleanup delete" looks identical to a real "delete" event:
Cleanup Delete:
{
RequestType: 'Delete',
ServiceToken: 'arn:aws:lambda:us-east-2:1234567890:function:stackname-resname-J0LWT56QSPIA',
ResponseURL: 'https://cloudformation-custom-resource-response-useast2.s3.us-east-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-2%3A1234567890%3Astack/stackname/3cc80cf0-5415-11e8-b6dc-503f3157b0d1%7Cresnmae%7C15521ba8-1a3c-4594-9ea9-18513efb6e8d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180511T140259Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=AKISOMEAWSKEYID%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Signature=3abc68e1f8df46a711a2f6084debaf2a16bd0acf7f58837b9d02c805975df91b',
StackId: 'arn:aws:cloudformation:us-east-2:1234567890:stack/stackname/3cc80cf0-5415-11e8-b6dc-503f3157b0d1',
RequestId: '15521ba8-1a3c-4594-9ea9-18513efb6e8d',
LogicalResourceId: 'resname',
PhysicalResourceId: '2018/05/11/[$LATEST]28bad2681fb84c0bbf80990e1decbd97',
ResourceType: 'Custom::Resource',
ResourceProperties: {
ServiceToken: 'arn:aws:lambda:us-east-2:1234567890:function:stackname-resname-J0LWT56QSPIA',
VpcId: 'vpc-35512e5d',
SomeValue: '4'
}
}
Real Delete:
{
RequestType: 'Delete',
ServiceToken: 'arn:aws:lambda:us-east-2:1234567890:function:stackname-resname-J0LWT56QSPIA',
ResponseURL: 'https://cloudformation-custom-resource-response-useast2.s3.us-east-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-2%3A1234567890%3Astack/stackname/3cc80cf0-5415-11e8-b6dc-503f3157b0d1%7Cresname%7C6166ff92-009d-47ac-ac2f-c5be2c1a7ab2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180524T154453Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKISOMEAWSKEYID%2F20180524%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Signature=29ca1d0dbdbe9246f7f82c1782726653b2aac8cd997714479ab5a080bab03cac',
StackId: 'arn:aws:cloudformation:us-east-2:123456780:stack/stackname/3cc80cf0-5415-11e8-b6dc-503f3157b0d1',
RequestId: '6166ff92-009d-47ac-ac2f-c5be2c1a7ab2',
LogicalResourceId: 'resname',
PhysicalResourceId: '2018/05/11/[$LATEST]c9494122976b4ef3a4102628fafbd1ec',
ResourceType: 'Custom::Resource',
ResourceProperties: {
ServiceToken: 'arn:aws:lambda:us-east-2:1234567890:function:stackname-resname-J0LWT56QSPIA',
VpcId: 'vpc-35512e5d',
SomeValue: '0'
}
}
The only interesting request field that I can see is the physical resource ID is different, but I don't know what to correlate that to, to detect if it is the real delete or not.
The problem seems to be the sample implementation of the sendResponse() function that is used to send the custom resource completion event back to CloudFormation. This method is responsible for setting the custom resource's physical resource ID. As far as I understand, this value represents the globally unique identifier of the "external resource" that is managed by the Lambda function backing the CloudFormation custom resource.
As can be seen in the CloudFormation's "Lambda-backed Custom Resource" sample code, as well as in the cfn-response NPM module's send() and the CloudFormation's built-in cfn-response module, this method has a default behavior for calculating the physical resource ID, if not provided as a 5th parameter, and it uses the CloudWatch Logs' log stream that is handling logging for the request being processed:
var responseBody = JSON.stringify({
...
PhysicalResourceId: context.logStreamName,
...
})
Because CloudFormation (or the AWS Lambda runtime?) occasionally changes the log stream to a new one, the physical resource ID generated by sendResponse() is changing unexpectedly from time to time, and confuses CloudFormation.
As I understand it, CloudFormation managed entities sometimes need to be replaced during an update (a good example is RDS::DBInstance that needs replacing for almost any change). CloudFormation policy is that if a resource needs replacing, the new resource is created during the "update stage" and the old resource is deleted during the "cleanup stage".
So using the default sendResponse() physical resource ID calculation, the process looks like this:
A stack is created.
A new log stream is created to handle the custom resource logging.
The backing Lambda function is called to create the resource and the default behavior set its resource ID to be the log stream ID.
Some time passes
The stack gets updated with new parameters for the custom resource.
A new log stream is created to handle the custom resource logging, with a new ID.
The backing Lambda function is called to update the resource and the default behavior set a new resource ID to the new log stream ID.
CloudFormation understands that a new resource was created to replace the old resource and according to the policy it should delete the old resource during the "cleanup stage".
CloudFormation reaches the "cleanup stage" and sends a delete request with the old physical resource ID.
The solution, at least in my case where I never "replace the external resource" is to fabricate a unique identifier for the managed resource, provide it as the 5th parameter to the send response routine, and then stick to it - keep sending the same physical resource ID received in the update request, in the update response. CloudFormation will then never send a delete request during the "cleanup stage".
My implemenation (in JavaScript) looks something like this:
var resID = event.PhysicalResourceId || uuid();
...
sendResponse(event, context, status, resData, resID);
Another alternative - which would probably only make sense if you actually need to replace the external resource and want to adhere to the CloudFormation model of removing the old resource during cleanup - is to use the actual external resource ID as the physical resource ID, and when receiving a delete request - to use the provided physical resource ID to delete the old external resource. That is what CloudFormation designers probably had in mind in the first place, but their default sample implementation causes a lot of confusion - probably because the sample implementation doesn't manage a real resource and has no update functionality. There is also zero documentation in CloudFormation to explain the design and reasoning.
It’s important to understand the custom resource life cycle, to prevent your data from being deleted.
A very interesting and important thing to know is that CloudFormation
compares the physical resource id you returned by your Lambda function
to the one you returned previously. If the IDs are different,
CloudFormation assumes the resource has been replaced with a new
resource. Then something interesting happens.
When the resource update logic completes successfully, a Delete
request is sent with the old physical resource id. If the stack update
fails and a rollback occurs, the new physical resource id is sent in
the Delete event.
You can read more here about custom resource life cycle and other best practices
Scenario:
I have a serverless/cloudformation script that re-deploys the same code with different configurations to AWS as lambdas and exposes each lambda via API Gateway.
So far the only way I've been able to do this is via copious amounts of copy and paste within the same script.. but its starting to drive me up the walls... thus, as I'm a complete newby to AWS and, navigating the AWS docs and internet has yielded pretty bad results, in answering this... I'm trying my luck here.
Within a cloudformation script:
"Resources":{
"LambdaResourceNG":{
"Type":"AWS::Serverless::Function",
"Properties":{
"Handler":"some-handlername::foo::bar",
"Runtime":"dotnetcore2.0",
"Environment":{
"Variables":{
"PictureOptions__OriginalPictureSuffix":{
"Fn::Join":[
"",
[
"_",
"ng",
"_",
{
"Fn::FindInMap":[
"Environments",
{
"Ref":"EnvironmentValue"
},
"PictureOptionsOriginalPictureSuffix"
]
}
]
]
},
},
"Events":{
"Bar":{
"Type":"Api",
"Properties":{
"Path":"/ng/bar",
"Method":"POST"
}
},
"Foo":{
"Type":"Api",
"Properties":{
"Path":"/ng/foo",
"Method":"POST"
}
}
}
}
},
}
Question:
In the script block above.. The resource is called LambdaResourceNG. If I wanted to have another resource...LambdaResourceKE... with all appropriate sections changed to KE. How would I make a "function" that I could re-use within this erm... language?
I've already found out how to use maps to swap out variables based on some env value... but how would one go about creating reusable blocks of code/config?
If the existing CloudFormation nested stacks feature doesn't suffice and you need real programmability then the final CF template can be the output of a higher-level process.
There are tools available to create templates e.g. AWS Cloud Development Kit, Troposphere and cfndsl.
Another option would be to drive the creation of the final template from a CLI. It doesn't have to be particularly sophisticated, just something that includes a template engine (like jinja2 or handlebars). Then you can program the inclusion of reusable template fragments, dynamically inject values into those fragments, iterate over loops as necessary, and emit a final CloudFormation template (or a main template and set of nested templates).
You can nest a CloudFormation Stack within another using the AWS::CloudFormation::Stack resource type. Nested stacks cannot exist without their parent, deleting the parent stack will delete all nested stacks. Note that the TemplateURL must point to S3, and that is where the aws cloudformation package CLI command helps by uploading a local file there and replacing the URL in the template.
Cross-stack references also helps in modularizing templates. For example, a "database network" stack can export the subnet ids and other values for any future database stack to use. Note that modularization goes further than merging text, but declaring and managing the resources lifecycle relationships correctly.
Stacks can even be composed further and across different regions and accounts using StackSets. This may be quite helpful when managing applications provisioned per tenant or sub-organization. This is frequently the case in "self-service IT" that can be achieved using CloudFormation with other services like AWS Service Catalog and AWS Marketplace.
Nested stacks are clumsy in that you don't necessarily want an entire stack just for a single resource. CloudFormation Modules would solve this problem nicely (reference). You can even package multiple resources in a single module.
You can create reusable modules with pre-packaged properties, which:
Reduce boilerplate configuration
Enforce company-wide standards
Modules are deployed to CloudFormation Registry, where they can be versioned and used by anyone in your company. You can use Parameters in the module to pass in properties just like you would a standard AWS resource. You can then create the custom modules like this:
Resources:
LambdaResourceNG:
Type: YourCompany::LambdaApi::FooBarApi
Properties:
ApiName: ng
LambdaResource:
Type: YourCompany::LambdaApi::FooBarApi
Properties:
ApiName: ke
To create a reusable template in Cloudformation. There are couple of things you need to keep in mind
Use Nested stack : using nested stack you can create a small stack for each AWS Service ( i.e VPC,LoadBalancer ), which you can use in other projects
Use Parameters : Use perameters as much as possible
Use Conditions : AWS Cloudformation provide solution to add conditions, Using Conditions we can use same template to perform multiple tasks