We're building an API using AWS SAM. Build on the Lambda Node Template in CodeStar. Things were going well until our template.yml file became too big. Whenever the code is pushed and CloudFormation starts to execute the change set and create a stack for the SAM endpoints, it fails and rolls back to the last successful build.
It seems that we have too many resources that exceeds the CloudFormation limit per stack.
I tried splitting the template file and edited the buildspec to handle two template files and do two AWS CloudFormation package commands and added another artifact. But it didn't work either. As only the first template is recognized and only one stack is created.
I can't find a way to make an automated deployment that creates multiple stacks.
I'd appreciate some input into this and suggestions to handle such a scenario.
Thanks in advance.
You should try using the nested stacks pattern. Instead of splitting your current stack into multiple parallel stacks, you will create a parent stack that will in turn create multiple child stacks.
More information here.
AWS SAM (as of SAM v1.9.0) supports nested applications which map to nested CloudFormation stacks which gets around the 200 resource limit. (AWS::Serverless::Application transforms into a AWS::CloudFormation::Stack)
https://github.com/awslabs/serverless-application-model/releases/tag/v1.9.0
The main subject to see is what is the components you have in your sam template ? is there any dependencies ? is all Functions shares the same API Gateway or not ? is all functions access DynamoDB table ?
In my case, I split the SAM by API [ API Gateway + functions ( CRUD)] in a mono repo way, each folder contains its sam template.
If you have a shared service like Redis, or SNS, SQS, you can have a separate stack with the export import Feature to import the ARN of the service.
Related
How is it currently done the handling of multiple lambda functions for a single stack/application?
Considering a use case with more than one function is it better to stick all together in the same repository or have one for each?
Having a single repository for all the functions would be much easier for me coming from old/classic backend development with a single codebase for all the business logic, but moving on the AWS ecosystem means I can no longer "deploy" my entire business logic with a single command since I need to zip a single function and update the archive with the aws cli, and that is impossible with standard merge requests or pipeline due the impossibility of automation for these steps (every time it could be a different function or multiple ones).
From the other side, having e.g. 5 or 6 repositories one for each lambda alongside the ones for frontend and AWS stack would be very impractical to manage.
Bundle your different lambda functions together as a Cloudformation stack. Cloudformation allows you to create multiple AWS services, bridge them together as you wish. There are many tools you can use to achieve this. AWS Cloudformation, AWS SAM (serverless application model) or third party tools like serverless and Terraform. Base concept is known as Infrastructure as Code (IAC).
As per respositories, you can have a single repository per stack. (AWS SAM provides sample codes with a good directory structure) You can try sam init as an example.
Consider AWS Serverless Application Model for your development. It allows you to bash script build, package and deploy using sam cli based on the yaml template. SAM will figure out the diff in your code by itself (because it runs CloudFormation under the hood). It allows not only to combine several functions into one package, but also add API gateways, dynamoDB tables and so much more! Another cool feature is that your functions will appear as an integrated application in Lambda console so you can monitor them all at the same time.
TLDR: Is there a way to -- using a single Cloudformation template -- deploy a lambda function with code in S3 or ECR?
Say I have two Lambda functions.
Function A has code in an S3 bucket and relies on a Lambda Layer
Function B is a containerized function with the image in an ECR instance
Here's my deployment setup for the two functions:
function-a/
s3-bucket.template // CFN Stack template for S3 bucket
lambda-function.template // CFN Stack template for lambda function
deploy.sh // Script that creates S3 bucket template,
builds/uploads code,
and creates lambda function stack
function-b/
ecr.template // CFN Stack template for ECR instance
lambda-function.template // CFN Stack template for lambda function
deploy.sh // Script that creates ECR,
builds/uploads docker image,
and creates lambda function stack
Results: 4 Cloudformation stacks, 1 S3 bucket, 1 ECR, 2 Lambda functions
I find this amount of configuration setup for these two functions needlessly complex.
I understand that buckets and registries need to exist. But I don't want to explicitly define/deploy/manage them using extra build steps.
What else I looked at: I checked AWS SAM -- but SAM also doesn't absolve me from managing the code deployment myself. I used AWS CDK which actually abstracts this away. But for certain reasons I don't want to use CDK here atm. I do not want to use the Serverless framework.
I'm disappointed that most of the examples from the documentation of Cloudformation and SAM just end up creating buckets and registries manually. This doesn't seem like a scalable way to handle it for many environments. This isn't Infrastructure-as-Code.
Is there a simpler way?
The S3 bucket and ECR would be reused for future functionality. So I think of it as two shared resources (S3 code bucket and ECR) and then two new resources (the new Lambda functions).
Most likely you'll have a stack of shared items, things are used by everything but don't structurally change that much. Then another stack of application functions, which will likely change more often. Separating these two different types of things is a good idea.
I want to do the first approach listed here, where in order to trigger various lambda functions to run when a single S3 bucket is updated, I would put an SNS queue in between.
Currently I have each Lambda as a stack, and I would prefer to keep it that way, especially since I have separate pipeline stages I need to separate anyways. However, I want to be able to make them all share the same SNS queue. What would be the best way to do this?
From my thinking, the best way to approach would be to create an "sns queue stack" that creates the topic, then pass that topic into each lambda stack and subscribe the lambda functions that way, but I'm still unsure of the best way to deploy this sns queue stack.
This is most confusing to me in regards to using a deployment pipeline in the CDK. I have multiple pipeline stages, each with multiple deployment groups and multiple lambda stacks in said deployment groups. How should I add in this stack to ensure it is deployed properly?
One guess I have would be to add it into the very first stage in the very first deployment group before all other stacks and then it should work for every other stage, but I'm not sure if this is a way that would work.
We can use two approaches.
Multiple stacks in single CDK Project:
We have single CDK project with multiple stacks with in the same project. For example we have 1 stack with SNS topic and 1 stack for each lambda and its SNS subscription. We can use sns topic name accross stacks like documented here
const snsStack = new MySnsStack(app, 'my-sns-stack');
// each stack takes property topic as input, which behind the scenes perform cloudformation export and import.
new MyLambdaOne(app, 'Stack2', {
topic: snsStack.topic
});
new MyLambdaTwo(app, 'Stack2', {
topic: snsStack.topic
});
All we need to do is cdk deploy and stacks are arranged and deployed in proper sequence. i.e. sns stack first and rest of the lambda stacks next based on references.
Multiple CDK Projects :
We have 1 stack per cdk project. so, we have multiple CDK projects to maintain. We then have to manually export topic Arn using cfnOutput from the first stack and import topic arn in other stacks using Fn.ImportValue.
Then we need to run multiple deployes cdk deploy MySnsStack, cdk deploy MyLambdaStack , etc. separately. First the sns stack, rest of them in parallel.
I would like to automate setting up the collection of AWS Application Load Balancer logs using Sumo Logic as documented here:
https://help.sumologic.com/07Sumo-Logic-Apps/01Amazon_and_AWS/AWS_Elastic_Load_Balancer_-_Application/01_Collect_Logs_for_the_AWS_Elastic_Load_Balancer_Application_App
This involves creating a bucket, creating a Sumo Logic hosted collector with an S3 source, taking the URL of the collector source provided by Sumo Logic and then creating an SNS Topic with an HTTP subscription where the subscription URL is the one provided by the Sumo Logic source.
The issue with this is that the SumoLogic source URL is not known at synthesis time. The Bucket must be deployed, then the Sumlogic things created, then the SNS topic created.
As best I can figure, I will have to do this through separate invocations of CDK using separate stacks, which is slower. One stack to create the bucket. After deploying that stack, use the Sumo Logic api to create or affirm prior creation of the Sumo Logic hosted collector and source, another CDK deploy to create the SNS topic and HTTP subscription.
I was just wondering if anyone knew of a better way to do this, perhaps some sort of deploy time hook that could be used.
There are two ways(which I know of) in which you can automate the collection of AWS Application Load Balancer.
Using CloudFormation
Sumo Logic have a template that creates the Collection process for AWS Application Load Balancer which is part of the AWS Observability Solution. You can fork the repository and can create your own CloudFormation template after removing resources you do not require.
Sumo Logic also have a Serverless Application which auto enable Access logging for existing and new (which are created after application installation) load balancer. Example template which uses the application.
Using Terraform
As mentioned by Grzegorz, you can create a terraform script also.
Disclaimer: Currently employed by Sumo Logic.
You could try using a Custom Resource SDK Call to trigger a lambda that does what you want.
https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_custom-resources.AwsSdkCall.html
(I know this is not a perfect answer as it suggests to use another tool, yet I believe it fulfills the needs expressed in the question)
How about using Terraform?
sumologic_s3_source in Terraform is able to create the source at Sumo AND output its URL for other uses within Terraform - e.g. to set up AWS resources.
The docs on this even mention URL being one of the returned values:
url - The HTTP endpoint to use with SNS to notify Sumo Logic of new
files.
Disclaimer: I am currently employed by Sumo Logic.
I have a serverless app on AWS that I am deploying using Cloudformation. I deploy the pipeline first using a pipeline.yml file, which creates the 'pipeline' stack (which populates the repo with code from an s3 bucket) and then runs it. This then creates the 'dev' stack using a seperate YAML file called template.yml in the newly created repo which contains the infrastructure for the lambdas, dbs, and other resources contained to make this app work. After a review, a developer can then release this to the 'prod' environment. The reason it is split it out like this is so that the pipeline gets created once, and the dev/prod environment stacks can be initiated multiple times.
In it's current state, there are parameters hardcoded within the template.yml that are used in dev/prod stack creation. Problem is, the user has to manually change these hardcoded values in the file before stack creation. Is there any way the user can edit these parameters as usual in the UI where I create the pipeline stack, and these parameters would bubble into the app/environment stack creation?
If my question does not make sense, I can definitely help further clarify. Thanks!