I want to do the first approach listed here, where in order to trigger various lambda functions to run when a single S3 bucket is updated, I would put an SNS queue in between.
Currently I have each Lambda as a stack, and I would prefer to keep it that way, especially since I have separate pipeline stages I need to separate anyways. However, I want to be able to make them all share the same SNS queue. What would be the best way to do this?
From my thinking, the best way to approach would be to create an "sns queue stack" that creates the topic, then pass that topic into each lambda stack and subscribe the lambda functions that way, but I'm still unsure of the best way to deploy this sns queue stack.
This is most confusing to me in regards to using a deployment pipeline in the CDK. I have multiple pipeline stages, each with multiple deployment groups and multiple lambda stacks in said deployment groups. How should I add in this stack to ensure it is deployed properly?
One guess I have would be to add it into the very first stage in the very first deployment group before all other stacks and then it should work for every other stage, but I'm not sure if this is a way that would work.
We can use two approaches.
Multiple stacks in single CDK Project:
We have single CDK project with multiple stacks with in the same project. For example we have 1 stack with SNS topic and 1 stack for each lambda and its SNS subscription. We can use sns topic name accross stacks like documented here
const snsStack = new MySnsStack(app, 'my-sns-stack');
// each stack takes property topic as input, which behind the scenes perform cloudformation export and import.
new MyLambdaOne(app, 'Stack2', {
topic: snsStack.topic
});
new MyLambdaTwo(app, 'Stack2', {
topic: snsStack.topic
});
All we need to do is cdk deploy and stacks are arranged and deployed in proper sequence. i.e. sns stack first and rest of the lambda stacks next based on references.
Multiple CDK Projects :
We have 1 stack per cdk project. so, we have multiple CDK projects to maintain. We then have to manually export topic Arn using cfnOutput from the first stack and import topic arn in other stacks using Fn.ImportValue.
Then we need to run multiple deployes cdk deploy MySnsStack, cdk deploy MyLambdaStack , etc. separately. First the sns stack, rest of them in parallel.
Related
TLDR: Is there a way to -- using a single Cloudformation template -- deploy a lambda function with code in S3 or ECR?
Say I have two Lambda functions.
Function A has code in an S3 bucket and relies on a Lambda Layer
Function B is a containerized function with the image in an ECR instance
Here's my deployment setup for the two functions:
function-a/
s3-bucket.template // CFN Stack template for S3 bucket
lambda-function.template // CFN Stack template for lambda function
deploy.sh // Script that creates S3 bucket template,
builds/uploads code,
and creates lambda function stack
function-b/
ecr.template // CFN Stack template for ECR instance
lambda-function.template // CFN Stack template for lambda function
deploy.sh // Script that creates ECR,
builds/uploads docker image,
and creates lambda function stack
Results: 4 Cloudformation stacks, 1 S3 bucket, 1 ECR, 2 Lambda functions
I find this amount of configuration setup for these two functions needlessly complex.
I understand that buckets and registries need to exist. But I don't want to explicitly define/deploy/manage them using extra build steps.
What else I looked at: I checked AWS SAM -- but SAM also doesn't absolve me from managing the code deployment myself. I used AWS CDK which actually abstracts this away. But for certain reasons I don't want to use CDK here atm. I do not want to use the Serverless framework.
I'm disappointed that most of the examples from the documentation of Cloudformation and SAM just end up creating buckets and registries manually. This doesn't seem like a scalable way to handle it for many environments. This isn't Infrastructure-as-Code.
Is there a simpler way?
The S3 bucket and ECR would be reused for future functionality. So I think of it as two shared resources (S3 code bucket and ECR) and then two new resources (the new Lambda functions).
Most likely you'll have a stack of shared items, things are used by everything but don't structurally change that much. Then another stack of application functions, which will likely change more often. Separating these two different types of things is a good idea.
I have two CloudFormation templates: the first template has with my Lambda functions and, the second, an SQS queue.
In the first template, I export the Lambda ARN and, in the second, I export the SQS queue URL and ARN.
Also, in the first template, I import the SQS queue URL and ARN to pass the value to the Lambda code and give permission to Lambda to send a message to SQS, respectively. In the second template, I import the Lambda ARN to give permission to Lambda to send a message to that SQS queue.
My concern is about the circular dependency between import/export values that was created with this configuration. At the moment of the deploy, there were no errors, but I'm wondering if this is a bad configuration or not.
If you are using exports/imports the only way this can work at all is with conditions on the resources using the imports, and running at least of the stacks twice. That’s not ideal, and it will also take two steps to delete the stacks (you can’t delete a stack if another stack imports an export from that stack).
The better solution here is to put all the dependencies in one stack (better yet, put it all in one stack if you can). In your case you could create the SQS queue and export it’s info and do all the dependent resources in the lambda stack.
Aim : Use terraform to create an SNS topic in multiple regions.
Right now it lets me create a single SNS in the default region or one specific region i input.
Hurdles :
providers.tf doesnt support iterating the region via a list
sns topic creation in terraform doesnt take allow region arguments to be passed as well.
The only workaround i could do is do something found here -
Using Terraform to manage multiple AWS regions
But this involves a lot of code duplication and is not efficient or scalable.
Is there any efficient iterative approach to solve this problem ?
We're building an API using AWS SAM. Build on the Lambda Node Template in CodeStar. Things were going well until our template.yml file became too big. Whenever the code is pushed and CloudFormation starts to execute the change set and create a stack for the SAM endpoints, it fails and rolls back to the last successful build.
It seems that we have too many resources that exceeds the CloudFormation limit per stack.
I tried splitting the template file and edited the buildspec to handle two template files and do two AWS CloudFormation package commands and added another artifact. But it didn't work either. As only the first template is recognized and only one stack is created.
I can't find a way to make an automated deployment that creates multiple stacks.
I'd appreciate some input into this and suggestions to handle such a scenario.
Thanks in advance.
You should try using the nested stacks pattern. Instead of splitting your current stack into multiple parallel stacks, you will create a parent stack that will in turn create multiple child stacks.
More information here.
AWS SAM (as of SAM v1.9.0) supports nested applications which map to nested CloudFormation stacks which gets around the 200 resource limit. (AWS::Serverless::Application transforms into a AWS::CloudFormation::Stack)
https://github.com/awslabs/serverless-application-model/releases/tag/v1.9.0
The main subject to see is what is the components you have in your sam template ? is there any dependencies ? is all Functions shares the same API Gateway or not ? is all functions access DynamoDB table ?
In my case, I split the SAM by API [ API Gateway + functions ( CRUD)] in a mono repo way, each folder contains its sam template.
If you have a shared service like Redis, or SNS, SQS, you can have a separate stack with the export import Feature to import the ARN of the service.
I have a large template for CloudFormation that has hundreds of resources. All are successfully updated, during an update, except one: an SNS alarm topic.
When deploying the stack, I get no errors, but even if the topic is non-existent the topic is never created.
I'm not expecting anyone to be able to provide me with a solution, but I would simply like to know how to troubleshoot the problem. It would be helpful to get output from the deployment, but the events are so few and really don't reflect the amount of resources being updated/created that they rarely help finding out what goes wrong.
Validation of the template is also successful, but that's almost a given since deploying also succeeds.
Regarding troubleshooting live CloudFormation stacks in general, CloudFormation just recently added support for Change Sets, which should help you preview changes and troubleshoot potential issues with updates before you attempt to apply them to your running stack.
Regarding SNS topics specifically, creating an SNS topic from scratch using the AWS::SNS::Topic resource works correctly. However, if you are using a TopicName property in your SNS resource, make sure that the name is unique across your entire AWS Account, as noted in the documentation:
Resource names must be unique across all of your active stacks. If you reuse templates to create multiple stacks, you must change or remove custom names from your template.
So reusing a constant TopicName in a stack deployed multiple times could cause the issue you're describing.
Also, if you're attempting to update existing CloudFormation-created topics with added/removed subscriptions, note the following important notice in the documentation:
Important
After you create an Amazon SNS topic, you cannot update its properties by using AWS CloudFormation. You can modify an Amazon SNS topic by using the AWS Management Console.
As a potential workaround for adding/removing subscriptions to existing SNS topics via CloudFormation, there is a cloudformation-helpers library containing a Custom::SnsSubscription resource (example) that might help.