I'm trying to get this repo going: https://github.com/mydatastack/google-analytics-to-s3.
A link is provided to launch the AWS CloudFormation stack, its meant to be one click to launch the stack but it is no longer working because the S3 bucket containing the template is no longer active.
As a result I'm trying to launch the stack myself via sam deploy --guided --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM since all the resources for the stack are within the repo. I've added this lambda layer for the paramiko package referenced by collector-ga.yaml to fix this error .
Frustratingly, I'm not quite up and running yet, GlueConfigurationLambda, an AWS lambda function (line 691) failed to create:
Waiting for changeset to be created..
CloudFormation stack changeset
---------------------------------------------------------------------------------------------------------------------
Operation LogicalResourceId ResourceType Replacement
---------------------------------------------------------------------------------------------------------------------
+ Add GoogleAnalyticsCollectorSta AWS::CloudFormation::Stack N/A
ck
---------------------------------------------------------------------------------------------------------------------
Changeset created successfully. arn:aws:cloudformation:eu-central-1:XXXXXX:changeSet/samcli-deploy1628597635/4ee26e-46b5-4131-bdba-1b9fc34f99d6
Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y
2021-08-10 13:14:04 - Waiting for stack create/update to complete
CloudFormation events from changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus ResourceType LogicalResourceId ResourceStatusReason
---------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS AWS::CloudFormation::Stack GoogleAnalyticsCollectorStack -
CREATE_IN_PROGRESS AWS::CloudFormation::Stack GoogleAnalyticsCollectorStack Resource creation Initiated
CREATE_FAILED AWS::CloudFormation::Stack GoogleAnalyticsCollectorStack Embedded stack
arn:aws:cloudformation:eu-central-1:
XXXXXX:stack/GAN2S3-GoogleAnal
yticsCollectorStack-JUATDT3EBD82/e19
a4950-ff27-11ea-943e-06072e1f2808
was not successfully created: The
following resource(s) failed to
create: [GlueConfigurationLambda].
Full Trace - https://pastebin.pl/view/50b3e402
My first question is if there's anywhere to get a more in-depth log of the error?
My second question is if anyone knows how to fix this error.
Can you have a look at the AWS Console CloudFormation application? You should be able to opt to view the Deleted stacks, after which you should be able to select the substack that has failed. In the events list of that deleted stack, you should be able to view a more precise error of what went wrong.
If it's still unclear from that precise error, feel free to edit the question to add the specific error and add a comment to this answer to draw my attention to it.
(Edit)
I've looked through the template file again and noticed the Lambda that's failing is still configured to use Node.js 8, which has been deprecated for some time. You should change it to a newer version, e.g., Node.js 14.
Find the currently supported runtimes here: Lambda runtimes
Related
I am trying to run Deploy multiple machine learning models for inference on AWS Lambda and Amazon EFS, I have done all the steps correctly, however, when I use the sam build --use-container command, I was falling into memory error.
In order to solve this problem I resizing EBS volume manually to 30 Gib and I used the sam build --debug --use-container command because I realized that cloud9 and my runtime had different versions of python.
After much research and trying I was able to build it successfully. But, when I went to deploy, using the command sam deploy --guided, I got the following:
CREATE_FAILED AWS::Lambda::Function MLModelUploadFunction Resource handler returned message:
"'MemorySize' value failed to
satisfy constraint: Member must
have value less than or equal to
3008 (Service: Lambda, Status Code:
400, Request ID: 3503cc50-aa63-45ca
-837a-e66085f8e215)" (RequestToken:
c2277c5f-0fa3-cd7f-df01-6c7e19d1ad0
a, HandlerErrorCode:
InvalidRequest)
CREATE_FAILED AWS::EC2::NatGateway NatGatewayA Resource creation cancelled
ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack sam4-app The following resource(s) failed to
create: [NatGatewayA,
MLModelUploadFunction]. Rollback
requested by user.
DELETE_IN_PROGRESS AWS::EC2::SubnetRouteTableAssociati RouteTableAssociationAPrivate -
on
DELETE_IN_PROGRESS AWS::IAM::Role
.
.
.
InternetGateway -
DELETE_COMPLETE AWS::EC2::InternetGateway InternetGateway -
DELETE_COMPLETE AWS::EC2::VPC EfsLambdaVpc -
ROLLBACK_COMPLETE AWS::CloudFormation::Stack sam4-app -
-----------------------------------------------------------------------------------------------------------------------------------------------------
Error: Failed to create/update the stack: sam4-app, Waiter StackCreateComplete failed: Waiter encountered a terminal failure state: For expression "Stacks[].StackStatus" we matched expected path: "ROLLBACK_COMPLETE" at least once
Researching about this error, I noticed that it is a problem with the lambda function. When I access Applications in the AWS Lambda it shows a creation error.
I tried various solutions like directly specifying the s3 bucket, using another version of sam, I tried running the MLInference-Lambda-EFS code, but nothing works.
Does anyone know how I can solve this problem? I have been struggling with this project for a long time.
Please run sam delete --stack-name aws-sam-cli-managed-default to remove the broken managed stack before going through guided deploy or --resolve-s3 again.
I ran the deploy command again sam deploy --s3-bucket test-bucket --guided and it worked when I specified a bucket name. This bucket is ignored by SAM and the stack is created without problem.
I am trying to deploy AWS API Gateway Developer Portal using serverless repo
The Cloud formation fails and triggers rollback of the deployment. The error that causes the failure is as follows:
Logical ID: StaticAssetUploader
Status: CREATE_FAILED
Reason: CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [ea5b3458-f29c-4950-b068-d0a3f352ad5f]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version.
Help is appreciated
I have migrated existing AWS Resources from one Cloudformation (CFT) stack to another CFT stack using below link.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-new-stack.html
After migration, my new CFT stack's status was "IMPORT_COMPLETE". Then I have created an AWS CodePipeline wherein my source is AWS CodeCommit and I am trying to deploy it in Cloudformation stack using CodePipeline.
In my CodePipeline I am using my new CFT stack where I have migrated my existing AWS resources and in the same template I have updated my code by added SQS queue policy and uploaded the code in CodeCommit.
So, when my AWS CodePipeline is getting triggered it is getting failed with "InternalFailure" error and it is not giving any specific error about why it is getting failed.
Also, I have checked into CloudTrail logs and there I can see my pipeline is getting failed after "UploadArchive" event which belongs to CodeCommit and it is nor moving further. Also, I tried to give administrator permission to my pipeline service role as well as cloudformation role but still the error is same.
Later, one thing I observed and that is when I update my new Cloudformation stack using AWS Cloudformation console then my stack's status is changing to "Update_Complete" status. Then after that if I try to update the code into CodeCommit then my pipeline is getting completed successfully.
So, not sure why my Pipeline is getting failed with "InternalFailure" when my stacks status is "IMPORT_COMPLETE". Could you please help me to understand if I am missing any specific step die to which my pipeline is getting failed with this error when my CFT stacks status is "IMPORT_COMPLETE" status
It's a bug in codepipeline. I'd recommend submitting at ticket to them in hopes they make a fix. I only found this out via support myself.
Context
I feel like I need to provide a lot of context here for the full scope of the problem to be understood, so apologies in advance if this goes a bit long or ends up providing too much information, I just want to ward off as many follow-up questions and clarification requests as I can.
I've got a project that's a tech handoff. It used to be in production under a different owner in a different AWS account. I'm trying to re-launch it in an AWS account I control and one of the packages is creating some problems for me.
It uses Serverless to provision a couple S3 buckets and their access policies, a couple IAM roles, and a bunch of ApiGateway methods. The package relies on nested stacks to get around the 200 resource limit as described here.
Finally, the IAM user that CircleCI connects as has the AdministratorAccess policy attached.
Problem
I keep getting failures from CircleCI during this step in the build
node_modules/.bin/serverless deploy --verbose --stage develop --region us-east-1 --package ./.serverless
The exact nature of the failure seems to be inconsistent i.e., it doesn't always fail at the same spot. At some point a resource just fails to create and the whole process rolls back. Here are a couple examples of run failures in the log with +/- 5 lines, followed by the actual error reported by Serverless
Run 1
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod001VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod002VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod003VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod004VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod006Options
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncAbcNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncDefNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncGhiNestedStack
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - DELETE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethod006Options
Serverless Error ---------------------------------------
An error occurred: FuncAbcNestedStack - Embedded stack arn:aws:cloudformation:us-east-1:ACCOUNT_ID:stack/org-package-develop-FuncAbcNestedStack/RESOURCE-ID-001 was not successfully created: The following resource(s) failed to create: [AbcLambdaFunction]. .
Run 2
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod001VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod002VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod005VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod006Options
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod004VarOptions
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncDefNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncGhiNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncAbcNestedStack
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - DELETE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethod001VarOptions
Serverless Error ---------------------------------------
An error occurred: FuncDefNestedStack - Embedded stack arn:aws:cloudformation:us-east-1:ACCOUNT_ID:stack/org-package-develop-FuncDefNestedStack/RESOURCE-ID-002 was not successfully created: The following resource(s) failed to create: [DefLambdaFunction]. .
Note: All the unique identifiers in the above logs have been replaces/obfuscated by new identifiers which are unique across both logs, not per log i.e., FuncAbcNestedStack appears in both logs because it exact same resource in the configuration.
Question
Given all the above, my question at this point is how do I debug this? This represents all the detail I (believe) is available to me in that I can't dive deeper to find out why a resource failed to create. I've read a bit about troubleshooting errors but nothing there has been terribly helpful since I'm not actually using EC2 directly.
Apr 4 Update
I've done a good amount of work trying to debug the templates. Mind you, I'm generally not working with the templates themselves, Serverless generates them and dumps them into an S3 bucket before they're applied.
Here are some steps I've taken
Updated to most recent version of Serverless (1.67.0, from 1.30.3)
Nuked existing stacks
Nuked related S3 bucket
Updated node runtime (12.16.1, from 8.10.0)
Downloaded and linted the CFN template that contains the failing lambda - no issues reported
I'm still getting the same results. When I re-run the build and check the CloudFormation event logs, I do see that a stack fails to create because a Lambda function within it fails to create. There's nothing special about this function (other Lambdas create successfully earlier in the run) other than the fact that it's the authorizer for every other function in the API, which may or may not be significant. I still can't find further detail as to why the lambda fails to create.
Apr 6 Update
Ok, now that I understand how the CloudFormation console works, here is now what I think is the undermost-lying error message
Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 0507722d-46e7-4340-bc68-fdba1ed469d6)
Looking at the CFN template itself for this nested stack, I now see what is going on. Every single lamba gets its own stack, and each stack across the entire package is compiled into a single ZIP file, whose size ends up being about 270MB or 20MB larger than the limit specified above. From this point, it seems like I have two possible paths forward
Figure out how to split the functions across multiple ZIPs
Change the webpack configuration so the compiled files are less bloated (I seriously don't know what's going on here - a 1k TypeScript file is coming out as 6.5MB after webpack)
You'll need to look at the nested stacks themselves. The AbcLambdaFunction and DefLambdaFunction resources should have more detailed failure stack events in the nested stacks than in the parent stacks. You'll likely need to fix AbcLambdaFunction and DefLambdaFunction in the nested stack templates, as the inconsistency is likely just due to whichever resource happened to fail first and started the rollback
If it's been a while since those templates have been run, it's likely Lambda Runtimes have been deprecated. The CloudFormation Linter should be able to check your templates for this and more possibilities
AWS Lambda limits are likely as well, I'd recommend trying things like this
Check to see if there any CloudWatch logs as well
I am using sam deploy to deploy lambda function and API gateway. It works fine but it doesn't work after I manually deleted the lambda function via AWS console. I got below error:
"ResourceStatusReason": "Function not found:
arn:aws:lambda:ap-southeast-2:286334053171:function:polaroid (Service:
AWSLambdaInternal; Status Code: 404; Error Code: ResourceNotFoundException;
Request ID: b431cbfc-7772-11e9-8022-1b92fa2cfa9e)
What is the proper way to delete the lambda and do a refresh deployment? If this happens, how can I force SAM to create the missing lambda function?
My lambda in template yaml looks like:
...
Resources:
PolaroidFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: test
CodeUri: ./lambdas
Handler: lib/index.fun
Runtime: nodejs8.10
Events:
polaroid:
Type: Api
Properties:
Path: /test
Method: post
...
I guess you already learnt the hard way that you should never manually delete resources managed by SAM or CloudFormation.
In general, if you just want to change the function, you can just call sam build and sam deploy, and the new version of it will be deployed. There is no need to delete anything. If you need a more advanced workflow, you will need to read blog posts. There is no one right way to do this.
To fix your immediate problem however, here is what you can do.1
Firstly, you need to get the generated AWS CloudFormation template:
▶ aws cloudformation get-template --stack-name HelloWorld \
--template-stage Processed --query TemplateBody | cfn-flip -y > processed.yml
Next, you need to comment out the function in the processed.yml file you just created, and also comment out the Lambda Permissions that refer to it. Save a backup of the original processed.yml file too.
Also, update any other template references to it if possible with the actual values CloudFormation computed when you built the stack, by getting them from your AWS console. For example, if you had references to ${HelloWorldFunction.Arn} you might have to update those references in the template with a string like arn:aws:lambda:ap-southeast-2:123456789012:function:HelloWorld-HelloWorldFunction-1NJGQI7GEAUM1.
Next, validate the template using AWS CloudFormation commands:
▶ aws cloudformation validate-template --template-body file://processed.yml
{
"CapabilitiesReason": "The following resource(s) require capabilities: [AWS::IAM::Role]",
"Description": "sam-app\nSample SAM Template for sam-app\n",
"Parameters": [],
"Capabilities": [
"CAPABILITY_IAM"
]
}
Next, you will update the stack using this modified template. By updating the stack this way, you get your template and real state to be back in sync from CloudFormation's point of view:
▶ aws cloudformation update-stack --template-body file://processed.yml --stack-name HelloWorld --capabilities CAPABILITY_IAM
{
"StackId": "arn:aws:cloudformation:ap-southeast-2:885164491973:stack/HelloWorld/af2c6810-7884-11e9-9bb3-068b1a8e1450"
}
If all goes well, your stack goes into UPDATE_COMPLETE state. Great!
Finally, uncomment all the resources you commented out, and restore all the original values. Then update stack a second time, and your stack should be restored to its original state.
See also:
AWS Knowledge Base, 2016, 2019, How do I update an AWS CloudFormation stack that's failing because of a resource that I manually deleted?.
More on the cfn-flip utility, if you don't have it.
1 Note that I tested this method using the default HelloWorld Python 2.7 example that ships with SAM.
I had a similar issue. In my case I had deleted the Lambda as an experiment while trying to reset the TRIM_HORIZON to get it to reprocess old events in a DynamoDB Stream.
I found a simpler solution:
Go into the CloudFormation Console and delete the deployed Stack.
sam deploy works fine again after that.
So as suggested in other answers I deleted the function manually from the console.
I was deploying the stack from CDK
The solution
comment the lambda function code (in cdk) of the function I deleted manually.
Deploy stack
Un-comment the code and deploy again
If you want to avoid deleting the stack and deploying it back again, or avoid aligning the CloudFormation template file, perhaps you can just align the resources in AWS to the template file.
That means, if you deleted a certain Lambda (for example) that was created initially from the template file, just create the same Lambda MANUALLY in AWS (either GUI or aws cli).
Now run 'sam deploy' again - you should be aligned.
Now remove the Lambda definition from the template file and deploy again - the Lambda should be removed and the CloudFormation will be aligned.