I create CloudFormation template for our resources, it includes Lambda functions, API Gateways, Roles, etc. For verifying our template I create CloudFormation stack using it, check some resources which I updated and after that I delete stack. But last time I got such message when I tried to delete stack:
CloudFormation is waiting for NetworkInterfaces associated with the
Lambda Function to be cleaned up.
I tried to stop deletion process and restart it, but I faced the same issue again. What is the problem and how can I fix that?
This is a well known issue. There are couple of things you can do.
1) Wait for the deletion to fail. Then try to delete. It should show you a checkbox to skip NetworkInterface. Select that.
2) Go to EC2-->NetworkInterfaces and detach/delete the NIC that was used by your resources. Then delete your CFT stack.
Related
Couldn't delete lambda stack via aws cdk:
Lambda was unable to delete lambda because it is a replicated function.
When deleting a CloudFormation Stack with CloudFront resource and associated Lambda#edge - CloudFormation first initiate a delete request for the CloudFront resource and the links to lambda#edge, since the lambda#edge is replicated to edges locations it takes up to a few hours to complete the links deletion.
My workaround is to run the complete clean-up in two phases -
Delete the stack and ignore the DELETE_FAILED status if just the lambda was failed to be deleted (and the other resources were deleted successfully)
Re-delete the DELETE_FAILED stack after a few hours (Or do it automatically using a cleanup lambda triggered by cron event to clean-up the CloudFormation stacks with DELETE_FAILED status)
If you are able to get in such a situation then yo most likely have freshly deleted a stack that did have lambda edge functions. In that case replicated functions get revealed (you dont see them while the edge function is inplace/being used). You dont need to do anything, just maybe to be patient - the replicated function will disappear after some time.
I am running a stack on cloudformation that creates some resources like Route53, etc...
I want to be able to recreate only some of the resources with the same value.
for example, One of the stack events creates image on ECR and I want to rebuild it. Run rollback on that event and then create it again with the same parameters.
How can I do that?
It is not possible to specify parts of a stack to 'rebuild'.
For some resources, you can modify an attribute to trigger a redeployment. The documentation will say Update requires: Replacement.
For other resources, you could:
Remove the resource from the template file
Update the stack with the template, which will cause CloudFormation to attempt to remove the resource (if it still exists)
Restore the template to the previous contents
Update the stack again, which will cause CloudFormation to deploy the 'new' resources
I wanted to update my stack. The stack failed with error Function not found: arn:aws:lambda....
And stack in status UPDATE_ROLLBACK_IN_PROGRESS more than 5 hours. How do I stop this process?
If you deleted the function outside of CloudFormation, then you can manually create a new function of the same name. This sometimes helps.
You can also wait till the rollback timeouts. And it usually does after a while, but the time varies.
Another reason why it gets stuck in this state could be due to nested stacks:
Nested Stacks are Stuck in UPDATE_COMPLETE_CLEANUP_IN_PROGRESS, UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS, or UPDATE_ROLLBACK_IN_PROGRESS
In this case a recommended option is indeed to contact support:
To fix the stack, contact AWS customer support.
Recent AWS blog post also describes the issue and possible solutions:
Why is my AWS CloudFormation stack stuck in the state CREATE_IN_PROGRESS, UPDATE_IN_PROGRESS, UPDATE_ROLLBACK_IN_PROGRESS, or DELETE_IN_PROGRESS?
Regarding the time to wait, the timeout varies:
In most situations, you must wait for your AWS CloudFormation stack to time out. The timeout length varies, and is based on the individual resource stabilization requirements that AWS CloudFormation waits for to reach the desired state.
In our case, we have mistakenly passed wrong image name to cloudformation template. After realising the mistake, we tried to stop the stack update, which made the stack stuck for forever in UPDATE_ROLLBACK_IN_PROGRESS status. SO during ECS service creation it got stuck.
Solution:
in Stack event check in which step is in progress. (our case ECS service update)
Go to ECS service.
Click on Update service.
Choose older task definitions.
And Update.
Your Task definition is reset to previous version. And roll back will complete successfully.
I tried changing my iam group names in cloud formation, and the new ones got created, but the old ones didn't get deleted. Is this expected? Should I have to remove these manually? Seems like it should get rid of the old ones automatically. Checking the stack information in the aws console says everything completed successfully.
Thanks
The Delete stack operation is a best effort operation. It deletes whatever resources it can. On retry after delete failed gives a warning window listing resources which require manual deletion. So if delete failed on your IAM groups first time, you will have to delete them manually. This explanation is only out of experience as I have seen such behavior many times. I do not now if such behavior is documented somewhere.
Usecase
I have cloudformation template with resources and lambda functions . Usually the cloudformation will rollback the stack when there is failure in creating it.
But what if i had pushed some resources or lambda functions with wrong logic or with less configurations . In such case i want to rollback the stack to previous working stack.
I Know i have rollback triggers to monitor my stack and trigger alarms. But i want to do it manually because, what if i had missing that test scenario and because of that the alarms not triggered.
Is it possible with cloudformation ?
Thanks
Any help is appreciated
No you can't do that with Cloudformation.
Your CD pipeline should keep track of any code builds and template configuration and you should rollback with old/archived code by pushing out a new stack with that old code.