AWS CloudFormation is stuck at CREATE_IN_PROGRESS - amazon-web-services

Question

I am building a fairly big REST API with AWS Lambda. Language is node.js. There are over 200 functions and few more to come. What EACH of these functions do is connect with a RDS database, get data or save data.
I am deploying this with aws sam tool. Below is the template.yaml. Please note I am posting just one method because from the outer look all methods looks the same except the endpoints they are pointing.
WSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
xxx-restapi
Sample SAM Template for xxx-restapi
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
VpcConfig:
SecurityGroupIds:
- sg-041f2459dcd921e8e
SubnetIds:
- subnet-038xxx2d
- subnet-c4dxxxcb
- subnet-af5xxxc8
Resources:
GetAllAccountingTypesFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: xxx-restapi/
Handler: source/accounting-types/accountingtypes-getall.getallaccountingtypes
Runtime: nodejs14.x
Events:
GetAllAccountingTypesAPIEvent:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /accountingtypes/getall
Method: get
GetAccountingTypeByIDFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: xxx-restapi/
Handler: source/accounting-types/accountingtypes-byid.getbyid
Runtime: nodejs14.x
Events:
GetAllAccountingTypesAPIEvent:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /accountingtypes/getbyid
Method: get
LambdaRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: root
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- ec2:DescribeNetworkInterfaces
- ec2:CreateNetworkInterface
- ec2:DeleteNetworkInterface
- ec2:DescribeInstances
- ec2:AttachNetworkInterface
Resource: '*'
Outputs:
# ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for functions"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
All of my methods are fine, they work as expected. However when I try to deploy, this get stuck at CREATE_IN_PROGRESS. But, if I reduce the number of functions and try, it works.
I examined the aws cloud trail logs, found something like below.
ErrorCode: Client.RequestLimitExceeded
Resources: [{"resourceType":"AWS::EC2::SecurityGroup","resourceName":"sg-041f245xxxxd921e8e"},{"resourceType":"AWS::EC2::Subnet","resourceName":"subnet-af5xxxc8"}]
and
ErrorCode: Client.DryRunOperation
Resources: [{"resourceType":"AWS::EC2::SecurityGroup","resourceName":"sg-041f2459xxxx1e8e"},{"resourceType":"AWS::EC2::Subnet","resourceName":"subnet-axxxx3c8"}]
There are multiple events like above. How can I fix this?

Answer 1

CloudFormation is probably creating too many functions at once, hence you're hitting a throttling limit. You could probably classify this as a bug in CloudFormation, so I think you should for sure report this to AWS or the CloudFormation team.
That being said, a possible workaround could be to deploy in steps. Adding some Lambda functions with each update. This is going to be a very cumbersome thing to do, but I don't see another way.
You could always make this process easier by using nested stacks (which you can then uncomment one by one). Maybe you could even circumvent the entire throttling limit with it, depending on how CloudFormation deals with that. But I'm not sure about this.
If you're managing so many resources in a single stack, you're also in danger of hitting other CloudFormation limits (especially since SAM abstracts multiple resources behind a single type). So using nested stacks could also prevent you from hitting those limits in the (near) future.