I am trying to run Deploy multiple machine learning models for inference on AWS Lambda and Amazon EFS, I have done all the steps correctly, however, when I use the sam build --use-container command, I was falling into memory error.
In order to solve this problem I resizing EBS volume manually to 30 Gib and I used the sam build --debug --use-container command because I realized that cloud9 and my runtime had different versions of python.
After much research and trying I was able to build it successfully. But, when I went to deploy, using the command sam deploy --guided, I got the following:
CREATE_FAILED AWS::Lambda::Function MLModelUploadFunction Resource handler returned message:
"'MemorySize' value failed to
satisfy constraint: Member must
have value less than or equal to
3008 (Service: Lambda, Status Code:
400, Request ID: 3503cc50-aa63-45ca
-837a-e66085f8e215)" (RequestToken:
c2277c5f-0fa3-cd7f-df01-6c7e19d1ad0
a, HandlerErrorCode:
InvalidRequest)
CREATE_FAILED AWS::EC2::NatGateway NatGatewayA Resource creation cancelled
ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack sam4-app The following resource(s) failed to
create: [NatGatewayA,
MLModelUploadFunction]. Rollback
requested by user.
DELETE_IN_PROGRESS AWS::EC2::SubnetRouteTableAssociati RouteTableAssociationAPrivate -
on
DELETE_IN_PROGRESS AWS::IAM::Role
.
.
.
InternetGateway -
DELETE_COMPLETE AWS::EC2::InternetGateway InternetGateway -
DELETE_COMPLETE AWS::EC2::VPC EfsLambdaVpc -
ROLLBACK_COMPLETE AWS::CloudFormation::Stack sam4-app -
-----------------------------------------------------------------------------------------------------------------------------------------------------
Error: Failed to create/update the stack: sam4-app, Waiter StackCreateComplete failed: Waiter encountered a terminal failure state: For expression "Stacks[].StackStatus" we matched expected path: "ROLLBACK_COMPLETE" at least once
Researching about this error, I noticed that it is a problem with the lambda function. When I access Applications in the AWS Lambda it shows a creation error.
I tried various solutions like directly specifying the s3 bucket, using another version of sam, I tried running the MLInference-Lambda-EFS code, but nothing works.
Does anyone know how I can solve this problem? I have been struggling with this project for a long time.
Please run sam delete --stack-name aws-sam-cli-managed-default to remove the broken managed stack before going through guided deploy or --resolve-s3 again.
I ran the deploy command again sam deploy --s3-bucket test-bucket --guided and it worked when I specified a bucket name. This bucket is ignored by SAM and the stack is created without problem.
Related
I'm trying to get this repo going: https://github.com/mydatastack/google-analytics-to-s3.
A link is provided to launch the AWS CloudFormation stack, its meant to be one click to launch the stack but it is no longer working because the S3 bucket containing the template is no longer active.
As a result I'm trying to launch the stack myself via sam deploy --guided --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM since all the resources for the stack are within the repo. I've added this lambda layer for the paramiko package referenced by collector-ga.yaml to fix this error .
Frustratingly, I'm not quite up and running yet, GlueConfigurationLambda, an AWS lambda function (line 691) failed to create:
Waiting for changeset to be created..
CloudFormation stack changeset
---------------------------------------------------------------------------------------------------------------------
Operation LogicalResourceId ResourceType Replacement
---------------------------------------------------------------------------------------------------------------------
+ Add GoogleAnalyticsCollectorSta AWS::CloudFormation::Stack N/A
ck
---------------------------------------------------------------------------------------------------------------------
Changeset created successfully. arn:aws:cloudformation:eu-central-1:XXXXXX:changeSet/samcli-deploy1628597635/4ee26e-46b5-4131-bdba-1b9fc34f99d6
Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y
2021-08-10 13:14:04 - Waiting for stack create/update to complete
CloudFormation events from changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus ResourceType LogicalResourceId ResourceStatusReason
---------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS AWS::CloudFormation::Stack GoogleAnalyticsCollectorStack -
CREATE_IN_PROGRESS AWS::CloudFormation::Stack GoogleAnalyticsCollectorStack Resource creation Initiated
CREATE_FAILED AWS::CloudFormation::Stack GoogleAnalyticsCollectorStack Embedded stack
arn:aws:cloudformation:eu-central-1:
XXXXXX:stack/GAN2S3-GoogleAnal
yticsCollectorStack-JUATDT3EBD82/e19
a4950-ff27-11ea-943e-06072e1f2808
was not successfully created: The
following resource(s) failed to
create: [GlueConfigurationLambda].
Full Trace - https://pastebin.pl/view/50b3e402
My first question is if there's anywhere to get a more in-depth log of the error?
My second question is if anyone knows how to fix this error.
Can you have a look at the AWS Console CloudFormation application? You should be able to opt to view the Deleted stacks, after which you should be able to select the substack that has failed. In the events list of that deleted stack, you should be able to view a more precise error of what went wrong.
If it's still unclear from that precise error, feel free to edit the question to add the specific error and add a comment to this answer to draw my attention to it.
(Edit)
I've looked through the template file again and noticed the Lambda that's failing is still configured to use Node.js 8, which has been deprecated for some time. You should change it to a newer version, e.g., Node.js 14.
Find the currently supported runtimes here: Lambda runtimes
Context
I feel like I need to provide a lot of context here for the full scope of the problem to be understood, so apologies in advance if this goes a bit long or ends up providing too much information, I just want to ward off as many follow-up questions and clarification requests as I can.
I've got a project that's a tech handoff. It used to be in production under a different owner in a different AWS account. I'm trying to re-launch it in an AWS account I control and one of the packages is creating some problems for me.
It uses Serverless to provision a couple S3 buckets and their access policies, a couple IAM roles, and a bunch of ApiGateway methods. The package relies on nested stacks to get around the 200 resource limit as described here.
Finally, the IAM user that CircleCI connects as has the AdministratorAccess policy attached.
Problem
I keep getting failures from CircleCI during this step in the build
node_modules/.bin/serverless deploy --verbose --stage develop --region us-east-1 --package ./.serverless
The exact nature of the failure seems to be inconsistent i.e., it doesn't always fail at the same spot. At some point a resource just fails to create and the whole process rolls back. Here are a couple examples of run failures in the log with +/- 5 lines, followed by the actual error reported by Serverless
Run 1
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod001VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod002VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod003VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod004VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod006Options
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncAbcNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncDefNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncGhiNestedStack
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - DELETE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethod006Options
Serverless Error ---------------------------------------
An error occurred: FuncAbcNestedStack - Embedded stack arn:aws:cloudformation:us-east-1:ACCOUNT_ID:stack/org-package-develop-FuncAbcNestedStack/RESOURCE-ID-001 was not successfully created: The following resource(s) failed to create: [AbcLambdaFunction]. .
Run 2
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod001VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod002VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod005VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod006Options
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod004VarOptions
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncDefNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncGhiNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncAbcNestedStack
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - DELETE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethod001VarOptions
Serverless Error ---------------------------------------
An error occurred: FuncDefNestedStack - Embedded stack arn:aws:cloudformation:us-east-1:ACCOUNT_ID:stack/org-package-develop-FuncDefNestedStack/RESOURCE-ID-002 was not successfully created: The following resource(s) failed to create: [DefLambdaFunction]. .
Note: All the unique identifiers in the above logs have been replaces/obfuscated by new identifiers which are unique across both logs, not per log i.e., FuncAbcNestedStack appears in both logs because it exact same resource in the configuration.
Question
Given all the above, my question at this point is how do I debug this? This represents all the detail I (believe) is available to me in that I can't dive deeper to find out why a resource failed to create. I've read a bit about troubleshooting errors but nothing there has been terribly helpful since I'm not actually using EC2 directly.
Apr 4 Update
I've done a good amount of work trying to debug the templates. Mind you, I'm generally not working with the templates themselves, Serverless generates them and dumps them into an S3 bucket before they're applied.
Here are some steps I've taken
Updated to most recent version of Serverless (1.67.0, from 1.30.3)
Nuked existing stacks
Nuked related S3 bucket
Updated node runtime (12.16.1, from 8.10.0)
Downloaded and linted the CFN template that contains the failing lambda - no issues reported
I'm still getting the same results. When I re-run the build and check the CloudFormation event logs, I do see that a stack fails to create because a Lambda function within it fails to create. There's nothing special about this function (other Lambdas create successfully earlier in the run) other than the fact that it's the authorizer for every other function in the API, which may or may not be significant. I still can't find further detail as to why the lambda fails to create.
Apr 6 Update
Ok, now that I understand how the CloudFormation console works, here is now what I think is the undermost-lying error message
Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 0507722d-46e7-4340-bc68-fdba1ed469d6)
Looking at the CFN template itself for this nested stack, I now see what is going on. Every single lamba gets its own stack, and each stack across the entire package is compiled into a single ZIP file, whose size ends up being about 270MB or 20MB larger than the limit specified above. From this point, it seems like I have two possible paths forward
Figure out how to split the functions across multiple ZIPs
Change the webpack configuration so the compiled files are less bloated (I seriously don't know what's going on here - a 1k TypeScript file is coming out as 6.5MB after webpack)
You'll need to look at the nested stacks themselves. The AbcLambdaFunction and DefLambdaFunction resources should have more detailed failure stack events in the nested stacks than in the parent stacks. You'll likely need to fix AbcLambdaFunction and DefLambdaFunction in the nested stack templates, as the inconsistency is likely just due to whichever resource happened to fail first and started the rollback
If it's been a while since those templates have been run, it's likely Lambda Runtimes have been deprecated. The CloudFormation Linter should be able to check your templates for this and more possibilities
AWS Lambda limits are likely as well, I'd recommend trying things like this
Check to see if there any CloudWatch logs as well
I'm trying to work my way through a Cloud Formation stack creation. The stack includes an AWS::CodeDeploy::Application using CodePlatform: Lambda. It also has an AWS::CodeDeploy::DeploymentGroup.
If I run create-stack without the deployment group present, everything seems to work. I can then go into the web UI, and add the deployment group by hand.
But if I describe the deployment group in the template, and run create-stack, the create of the deployment group fails, and the stack gets rolled back. The error message looks like:
For Lambda deployment group, ec2TagFilters can not be specified (Service: AmazonCodeDeploy; Status Code: 400; Error Code: InvalidEC2TagException; Request ID: c4347652-c755-11e8-b8f1-6f54b77ae7fe)
... but my template's description of the deployment group doesn't include Ec2TagFilters
Ran into the same thing. No reference to the parameter, still errors. Added a Null Ec2TagFilters parameter and no luck. Weirdly, I then added a Deployment Style parameter, tried again, and it succeeded. Here's the CFN I ended with:
TestSSHConnectivityCodeDeployGroup:
Type: AWS::CodeDeploy::DeploymentGroup
Condition: InPrimaryRegion
Properties:
ApplicationName: !Ref TestSSHConnectivityCodeDeployApplication
DeploymentGroupName: Lambda-TestSSHConnectivity
DeploymentConfigName: CodeDeployDefault.LambdaAllAtOnce
DeploymentStyle:
DeploymentOption: WITH_TRAFFIC_CONTROL
DeploymentType: BLUE_GREEN
ServiceRoleArn: !Sub arn:aws:iam::${AWS::AccountId}:role/AWS-CodeDeploy-Service
I had this error happening in only 1 of my stacks in us-west-2. It was only a gamma environment so I deleted the stack, re-created it and everything was fine.
That won't work for everyone, but it did work in my case.
I'm creating a Nodejs microservice for AWS Lambda. I scaffolded by project using AWS Codestar, and that set me up with a CI/CD pipeline that automatically deploys the lambda function. Nice.
The issue is that every time it deploys the lambda function it must delete and recreate the function, thus deleting any versions or aliases I made.
This means I really can't roll back to other releases. I basically have use git to actually revert the project, push to git, wait for the super-slow AWS Code Pipeline to flow through successfully, and then have it remake the function. To me that sounds like a pretty bad DR strategy, and I would think the right way to rollback should be simple and fast.
Unfortunately, it looks like the CloudFormation section of AWS doesn't offer any help here. When you drill into your stack on the first CloudFormation page it only shows you information about the latest formation that occurred. Dear engineers of AWS CloudFormation: if there was a page for each stack that showed a history of CloudFormation for this stack and an option to rollback to it, that would be really awesome. For now, though, there's not. There's just information about the latest formation that's been clouded. One initially promising option was "Rollback Triggers", but this is actually just something totally different that lets you send a SNS notification if your build doesn't pass.
When I try to change the CodePipeline stage for deploy from CREATE_CHANGE_SET to CREATE_UPDATE I then get this error when it tries to execute:
Action execution failed UpdateStack cannot be used with templates
containing Transforms. (Service: AmazonCloudFormation; Status Code:
400; Error Code: ValidationError; Request ID:
bea5f687-470b-11e8-a616-c791ebf3e8e1)
My template.yml looks like this by the way:
AWSTemplateFormatVersion: 2010-09-09
Transform:
- AWS::Serverless-2016-10-31
- AWS::CodeStar
Parameters:
ProjectId:
Type: String
Description: AWS CodeStar projectID used to associate new resources to team members
Resources:
HelloWorld:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: nodejs8.10
Environment:
Variables:
NODE_ENV: staging
Role:
Fn::ImportValue:
!Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]
Events:
GetEvent:
Type: Api
Properties:
Path: /
Method: get
PostEvent:
Type: Api
Properties:
Path: /
Method: post
The only options in the CodePipeline "Deploy" action are these:
It would be really great if someone could help me to see how in AWS you can make Lambda functions with CodePipeline in a way that they are easy and fast to rollback. Thanks!
I used AWS CodeStar to create a new application with the "Express.js Aws Lambda Webservice" CodeStar template. This was great because it set me up with a simple CI/CD pipeline using AWS CodePipeline. By default the pipeline has 3 steps for grabbing the source code from a git repo, running the build step, and then deploying to "dev" environment.
My issue is that I can't set it up so that my pipeline has multiple environments: dev, staging, and prod.
My current deploy step has 2 actions: GenerateChangeSet and ExecuteChangeSet. Here are the configurations for the actions in original dev environment build step which work great:
I've created a new deploy stage at the end of my pipeline to deploy to staging, but honestly I'm not sure how to change the configurations. I'm thinking ultimately I want to be able to go into the AWS Lambda section of the AWS console and see three independent lambda functions: binance-bot-dev, binance-bot-staging, binance-bot-prod. Then each of these I could set as cloudwatch scheduled events or expose with their own api gateway url.
This is the configuration that I tried to use for a new deployment stage:
I'm really not sure if this configuration is correct and what exactly I should change in order to deploy in the way I want.
For example, should I be changing "Stack name", or should I keep that as "awscodestar-binance-bot-lambda" or change it for each environment as I am here?
Also, I'm pointing to a different template.yml file in the project. The original template.yml looks like this:
AWSTemplateFormatVersion: 2010-09-09
Transform:
- AWS::Serverless-2016-10-31
- AWS::CodeStar
Parameters:
ProjectId:
Type: String
Description: AWS CodeStar projectID used to associate new resources to team members
Resources:
Dev:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: nodejs4.3
Environment:
Variables:
NODE_ENV: dev
Role:
Fn::ImportValue:
!Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]
Events:
GetEvent:
Type: Api
Properties:
Path: /
Method: get
PostEvent:
Type: Api
Properties:
Path: /
Method: post
For template.staging.yml I use the exact same config except I changed "Dev:" to "Staging:" under "Resources", and I also changed the value of the NODE_ENV environment variable. So, I'm basically wondering is this the correct configuration for what I'm trying to achieve?
Assuming that everything in the configuration is correct, I then need to troubleshoot this error. With everything set as described above I can run my pipeline, but when it gets to my staging build step the GenerateChage_Staging action fails with this error message:
Action execution failed User:
arn:aws:sts::954459734159:assumed-role/CodeStarWorker-binance-bot-CodePipeline/1524253307698
is not authorized to perform: cloudformation:DescribeStacks on
resource:
arn:aws:cloudformation:us-east-1:954459734159:stack/awscodestar-binance-bot-lambda-staging/*
(Service: AmazonCloudFormation; Status Code: 403; Error Code:
AccessDenied; Request ID: dd801664-44d2-11e8-a2de-8fa6c42cbf86)
It seem to me from this error message that I need to add the "cloudformation:DescribeStacks" for my "CodeStarWorker-binance-bot-CodePipeline" so I go to IAM -> Roles and click on the CodeStarWorker-binance-bot-CodePipeline role. However, when I click on "CodeStarWorker-binance-bot-CodePipeline" and drill into the policy information for CloudFormation it looks like this role already has permissions for "DescribeStacks"!
If anyone could point out what I'm doing wrong or offer any guidance on understanding and thinking about how to do multiple environments with AWS CodePipeline that would be great. thanks!
UPDATE:
I changed the "Stack name" in my Deploy_To_Staging pipeline stage back to "awscodestar-binance-bot-lambda". However, I then get this error form the GenerateChange_Staging action:
Action execution failed Invalid TemplatePath:
binance-bot-BuildArtifact::template-export.staging.yml. Artifact
binance-bot-BuildArtifact doesn't exist
UPDATE 2:
In the root of my project I have the buildspec.yml file that was generated by CodeStar. It looks like this:
version: 0.2
phases:
install:
commands:
# Install dependencies needed for running tests
- npm install
# Upgrade AWS CLI to the latest version
- pip install --upgrade awscli
pre_build:
commands:
# Discover and run unit tests in the 'tests' directory
- npm test
build:
commands:
# Use AWS SAM to package the application using AWS CloudFormation
- aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template template-export.yml
- aws cloudformation package --template template.staging.yml --s3-bucket $S3_BUCKET --output-template template-export.staging.yml
- aws cloudformation package --template template.prod.yml --s3-bucket $S3_BUCKET --output-template template-export.prod.yml
artifacts:
type: zip
files:
- template-export.yml
I then added this to the CloudFormation section:
Then I add this to the "build: -> commands:" section:
- aws cloudformation package --template template.staging.yml --s3-bucket $S3_BUCKET --output-template template-export.staging.yml
- aws cloudformation package --template template.prod.yml --s3-bucket $S3_BUCKET --output-template template-export.prod.yml
And I added this to the "files:"
template-export.staging.yml
template-export.prod.yml
HOWEVER, I am still getting an error that "binance-bot-BuildArtifact does not exist".
Here is the full error after making the buildspec.yml change:
Action execution failed Invalid TemplatePath:
binance-bot-BuildArtifact::template-export.staging.yml. Artifact
binance-bot-BuildArtifact doesn't exist
It seems very strange to me that I can access "binance-bot-BuildArtifact" in one stage of the pipeline but not another. Could it be that the build artifact is only available to the one pipeline stage directly after the build stage? Can someone please help me to be able to access this "binance-bot-BuildArtifact"? Thanks!
For example, should I be changing "Stack name", or should I keep that as "awscodestar-binance-bot-lambda" or change it for each environment as I am here?
You should use a unique stack name for each environment. If you didn't, you would be replacing your 'dev' environment with your 'staging' environment, and so forth.
So, I'm basically wondering is this the correct configuration for what I'm trying to achieve?
I don't think so. You should use the exact same template for each environment. In order to change the environment name for each of your deploys, you can use the 'Parameter Overrides' field to choose the correct value for your 'Environment' parameter.
it looks like this role already has permissions for "DescribeStacks"!
Could the issue here be that your IAM role only has DescribeStacks permission for the dev stack? It looks like it does not have permission to describe the staging stack. Maybe you can add a 'wildcard'/asterisk to the policy so that it matches all of your stack names?
Could it be that the build artifact is only available to the one pipeline stage directly after the build stage?
No, that has not been my experience with CodePipeline. Unfortunately I don't know why it's telling you that your artifact can't be found.
robrtsql has already provided some good advice in terms of using the same template in both stages.
You might find this walkthrough useful.
Basically, it describes adding a Cloudformation "template configuration" which allows you to specify parameters to the Cloudformation stack.
This will allow you to deploy the same template in both your dev and prod environments, but also allow you to tell the difference between a dev deployment and a prod deployment, by choosing a different template configuration in each stage.