I have an app with a lot of resources (a bunch of DynamoDB tables, lambda functions, etc) and apparently I've run into the hard limit of 200 resources. The specific error is:
Template format error: Number of resources, 204, is greater than the maximum allowed, 200
The error message is pretty clear, but I'd like to know what my options are. Worst case, I can split the app into several pieces. Are there any alternative strategies?
You can create nested stacks that would also have the advantage of simpler testing, improving re-use and using different roles.
Common practice is to separate out different layers into different stacks. For example, build the VPC in one stack, deploy back-end in another stack and the front-end in another stack.
See: Use Nested Stacks to Create Reusable Templates and Support Role Specialization
The resource limit is now 500 if that helps!
AWS CloudFormation now supports increased limits on five service quotas
I have to face the same problem in the serverless framework. What I do!
Create microservices for each module like Authentication, User-management, SMS Gateway, Notification etc. that helps to manage code and AWS resources.
At the end expose API to create AWS custom domain and assign cloud formation to it.
I follow this blog, it's help and Serverless also suggest flow link.
Related
How is it currently done the handling of multiple lambda functions for a single stack/application?
Considering a use case with more than one function is it better to stick all together in the same repository or have one for each?
Having a single repository for all the functions would be much easier for me coming from old/classic backend development with a single codebase for all the business logic, but moving on the AWS ecosystem means I can no longer "deploy" my entire business logic with a single command since I need to zip a single function and update the archive with the aws cli, and that is impossible with standard merge requests or pipeline due the impossibility of automation for these steps (every time it could be a different function or multiple ones).
From the other side, having e.g. 5 or 6 repositories one for each lambda alongside the ones for frontend and AWS stack would be very impractical to manage.
Bundle your different lambda functions together as a Cloudformation stack. Cloudformation allows you to create multiple AWS services, bridge them together as you wish. There are many tools you can use to achieve this. AWS Cloudformation, AWS SAM (serverless application model) or third party tools like serverless and Terraform. Base concept is known as Infrastructure as Code (IAC).
As per respositories, you can have a single repository per stack. (AWS SAM provides sample codes with a good directory structure) You can try sam init as an example.
Consider AWS Serverless Application Model for your development. It allows you to bash script build, package and deploy using sam cli based on the yaml template. SAM will figure out the diff in your code by itself (because it runs CloudFormation under the hood). It allows not only to combine several functions into one package, but also add API gateways, dynamoDB tables and so much more! Another cool feature is that your functions will appear as an integrated application in Lambda console so you can monitor them all at the same time.
I have an architectural question about the design and organisation of AWS Serverless resources using CloudFormation.
Currently I have multiple stack organised by the domain specific purpose and this works well. Most of the stack that contain Lambdas have to transformed using Serverless (using SAM for all). The async communication is facilitated using a combination of EventBridge and S3+Events and works well. The issue I have is with synchronous communication.
I don't want to reference Lambdas from other stacks using their exported names from other stacks and invoke them directly as this causes issues with updating and versions (if output exports are referenced in other stacks, I cannot change the resource unless the reference is removed first, not ideal for CI/CD and keeping the concerns separate).
I have been using API Gateway as an abstraction but that feels rather heavy handed. It is nice to have that separation but having to have domain and DNS resolving + having the API GW exposed externally doesn't feel right. Maybe there is a better way to configure API GW to be internal only. If you had success with this, could you please point me in the direction?
Is there a better way to abstract invocation of Lambda functions from different stacks in a synchronous way? (Common template patterns for CF or something along those lines?)
I see two questions:
Alternatives for Synchronous Lambda Functions with API Gateway .
Api Gateway is one easy way, with IAM Authentication to make it secure. HTTP Api is much simplified and cheaper option compared to REST APIs. We can choose Private Api rather than a Regional/Edge, which is not exposed outside VPC to make it even move secure.
we can have a private ALB with target as Lambda functions, for a simple use case that doesn't need any API gateway features.(this will cost some amount every month)
We can always call lambdas directly with AWS SDK invoke.
Alternatives to share resources between templates.
Exporting and Importing will be bit of problem if we need to delete and recreate the resource, shouldn't be a problem if we are just updating it though.
We can always store the Arn of the Lambda function in an SSM parameter in source template and resolve the value of the Arn from SSM parameter in destination template. This is completely decoupled. This is better than simply hard coding the value of Arn.
Is there a quick way to find out which regions have any resources in my account? I'm specifically using the AWS .NET SDK but the answer likely applies to other AWS SDKs and the CLI since they all seem to be just wrappers to the REST API. I can obviously run all the List* methods across all regions but I'm thinking there must be a more optimal way to decide whether to query the entire region or not. Maybe something in billing, but it also needs to be relatively up-to-date, maybe within the last 5 minutes or so. Any ideas?
There is no single way to list all resources in an AWS account or in multiple regions.
Some people say that Resource Groups are a good way to list resources, but I don't think they include "everything" in an account.
AWS Config does an excellent job of keeping track of resources and their history, but it is also limited in the types of resources it tracks.
My favourite way to list resources is to use nccgroup/aws-inventory: Discover resources created in an AWS account. It's a simple HTML/JavaScript file that makes all the 'List' calls for you and shows them in a nicely formatted list.
I've been learning more and more about AWS lately. I've been reading through the white papers and working my way through the various services. I've been working on PHP applications and front-end dev for a while now. Two things really stuck out to me. Those two things are server-less architecture using Lambdas with event-triggers and SQS (queues). The last three years I have been working with REST over HTTP with frameworks like Angular.
It occurred to me though that one could create an entire back-end/service layer through Lambda's and message queues alone. Perhaps I'm naive as I have never used that type of architecture for a real world project but it seems like a very simple means to build a service layer.
Has anyone built a web application back-end consisting of only Lambdas and message queues as opposed to "traditional" http request with REST. If so what types of drawbacks are there to this type of architecture besides relying so heavily on a vendor like AWS?
For example, wouldn't it be entirely possible to build a CMS using these technologies where the scripts create the AWS assets programmatically given a key with full admin rights to an account?
Yes, you can practically create the entire backend service using serverless architecture.
There are a lot of AWS services that usually play into the serverless gambit of things.
DynamoDB, SNS, SQS, S3 to name a few.
AWS Lambda is the backbone and sort of acts as a glue to bind these services.
Serverless doesn't mean you move away from "traditional" http request to message queues. If you need the web interface you would still need to use HTTP. You would primarily use message queues to decouple your services.
So, if you want the service to be accessible over HTTP just like your REST services and still be serverless then you can do that as well. And for that you will need to use AWS API Gateway in conjunction with AWS Lambda
One primary drawback/limitation is that debugging is not very straightforward. You cannot login to the system and cannot attach remote debuggers. And then obviously you get tied into the vendor.
Then there are limitations on the resources. E.g. Lambda can offer you a maximum memory footprint of 5GB, so if you need to do some compute intensive job that needs more memory and can't be broken down into sub tasks then serverless (AWS Lambda) is not an option for you.
I am using AWS to build an API, and deploy this to multiple stages.
When a call is made to a specific environment, I need to get a stage variable in Lambda and then data is recorded in a DynamoDB table such as "environment-Table".
Is this the best way to work with environments (like development, production etc) using AWS API Gateway, Lambda and DynamoDB?
It difficult to say what the best approach is for your specific situation, given the limited data in your post. Managing multiple environments such as development and production was one of the intended uses of stage and stage variables. I don't see any obvious problems with what your are proposing.
Depending on your use case, you can call a Lambda function to record data in DynamoDB, or you may be able to skip the Lambda function and record the data in DynamoDB directly using the AWS proxy integration type.