How can I force delete my AWS cloud formation stack? The tied resources were already deleted previously like the DNS, route53, etc. Whenever I delete the stack, it failed because it couldn't find the resources.
The specified hosted zone does not exist. DELETE_FAILED.
This made the deletion halt and made the stack still active. How can I force clean-up and remove this stack?
Based on this thread, it appears that stack deletion can be problematic if the stack is altered.
https://forums.aws.amazon.com/thread.jspa?threadID=71050&start=0&tstart=0
Looks like you need to go via Support to get it deleted.
That said, I have been able to delete stacks where I have manually altered the components. Sometimes you need to wait a couple of hours after you terminate instances, VPCs, subnets etc before the stack deletion actually succeeds.
AWS Cloud formation stack does not get deleted until all dependency resources either used in other stack or busy with non stack resources. So better see which resource is shared some where else and try to release that.
hope that help.
It seems the Route53 records were either deleted manually before cloudformation or there was some failure within cloudformation after it deleted the record. Its stuck because it can't find the hosted zone because it was already deleted if indeed this is what was described in your post. You can try to update the stack with the original script and then delete the stack via cloudformation without any manual intervention.
Check your IAM role and the policy, review the privileges that you have associated with the role, so that if there is any permission issue, that should resolve the issue.
Refer: https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-stack-delete-failed/
Need to check the reason for failure in stack details
Either delete this manually, or retain those resources while eleteing
eg:
$ aws cloudformation delete-stack --stack-name my-stack --retain-resources myresource1 myresource2
Related
Our team had an issue where someone manually modified an IAM role for an operational event. We hoped that a CloudFormation stack redeployment would revert the state of the IAM role, however, the manual change was still there.
My working theory is that since the IAM role arn is the same, that CloudFormation does not delete and recreate it. Is that accurate? And if so, how do we ensure all relevant resources are torn down during a deployment?
My working theory is that since the IAM role arn is the same, that CloudFormation does not delete and recreate it. Is that accurate?
CloudFormation (CFN) does not check for any changes made outside of its control. You could remove the role, and CFN would still "think" that the role is there.
If you change a resource created by CFN manually outside of CFN (bad practice), you have so called a stack drift. CFN by itself is not aware of any changes made to resources it creates that occurred outside its control. But, CFN provides special tools which you have to explicitly call to detect the drift:
Detect drift on an entire CloudFormation stack
Not all resources support drift detection, but AWS::IAM::Role is one which does.
And if so, how do we ensure all relevant resources are torn down during a deployment?
Not sure what do you mean here. But you have to manually fix the drift. You have four choices:
Change the role back to its original state,
Update template to reflect the external changes,
Use import to import the changed role to stack,
Delete the entire stack, and create new one from the original template.
The last choice ensures that the modified role is also deleted and recreated in its original form.
I need to automatically delete a cloudformation stack after the EC2 instance created using that stack has finished running its UserData. I have tried to run deletion from the ec2 instance but it gives me permission error as the ec2 instance itself is deleted before deleting the whole stack.
I quite don't understand what you want in your question. But in general, if you delete the CloudFormation stack, all the resources created by that stack will also be deleted. All your resources, including EC2, will be deleted first before you can see successful deletion of your CloudFormation stack.
Context: I have been using CloudFormation for provisioning application resources for a while, and that has worked out just fine. However, I recently moved my application over to different infrastructure (Kubernetes), and to go live with that change, I modified the Route53 DNS record to point to the new resources, and left up all the AWS resources created by CloudFormation. For clarity, the DNS record which I modified was created as part of the CloudFormation stack.
Problem: I want to delete the now unused resources, including the CloudFormation stack itself. However, doing so would either delete a DNS record that I very much care about, or fail to delete the DNS record because it has been modified, rolling back the entire operation.
Question: Does anybody know a clever way I can remove a CloudFormation stack while still preserving the Route53 entries created by it with zero downtime?
Note: I do not want to manually delete the resources created by CloudFormation, except for the DNS records, and leave the stack hanging around.
I recently had a need to preserve my Route53 records that were created with CloudFormation. I performed the following exercise based on #jarmod response and it worked perfectly. The key here is the: "DeletionPolicy" : "Retain" More information on the DeletionPolicy attribute. I'll also note that when I applied the change set to change my deletion policy, CloudFormation DID NOT detect any drift in the Route53 entry. Success!
Create a basic CloudFormation stack; Load Balancer, ASG and a Route53 record pointing to the Load Balancer
Change the Route53 entry to point to another location using the Route53 console
Apply the "DeletionPolicy" : "Retain" to my Route53 resource block via a change set
Delete the CloudFormation stack from the AWS console
Confirm Route53 still has your entries after the stack has been deleted
Image of CloudFormation Skipping Delete
Deny delete privileges for Route53 and delete the stack in 2 phases.
Delete Stack Fails
When stacks are in the DELETE_FAILED state because AWS CloudFormation
couldn't delete a resource, rerun the deletion with the
RetainResources parameter and specify the resource that AWS
CloudFormation can't delete. AWS CloudFormation deletes the stack
without deleting the retained resource. Retaining resources is
useful when you can't delete a resource, such as an S3 bucket that
contains objects that you want to keep, but you still want to delete
the stack. After you delete the stack, you can manually delete
retained resources by using their associated AWS service.
I have an AWS account where multiple EC2 instances, load balancers, target groups, security groups etc are setup by multiple owners.
We use terraform to set this up but sometimes due to corruption, the state becomes inconsistent. Current mechanism to recover is to manually destroy all resources in that account owned by a particular owner.
Is there an easy way to nuke all resources in an AWS account belonging to a particular owner?
There is no way to delete all resources in an account owned by a particular user but there is a way to delete all resources in an account.
You can use aws-nuke which was created somewhat out of the same use case you described.
At first, you need to set an account alias for your account.
You must create a config file.
Then you can list down all resources that will be deleted using the following command:
aws-nuke -c config/nuke-config.yml --profile aws-nuke-example
Add --no-dry-run option to permanently delete all resources in the same command.
There are also multiple filter options available such as target, resource type, exclude, etc. that you can leverage to suit your needs.
Agree with the other answer that there is no easy way delete orphan resources.
But I see the original issue is that the terraform state is corrupted.
You can checkout the terraform import feature which lets you generate state file from aws resources. In that way you can connect your config to resources again.
Short answer: no.
Longer answer: actually, that's also no. There's no built-in capabillity for this.
The case you're describing is not within the bounds of typical AWS usage... destroying everything in an account -- usually -- should not be easy.
Of course, you could script it, fairly trivially, by wrapping calls to aws-cli to custom code to iterate through the resources and generate additional requests to destroy them... but if you do, lock that code away, since such capability is inherently dangerous.
You can delete all your resources you created, you'll need to automate, see a sample here:
Creation
https://github.com/jouellnyc/AWS/tree/master/create_aws_vpc2
Deletion
https://github.com/jouellnyc/AWS/blob/master/create_aws_vpc2/delete_lb_and_vpc.sh
Other
I've had some success with cloud nuke (played around for a few min; not in depth):
https://github.com/gruntwork-io/cloud-nuke
I dont think there is any state forward way to do it but to check if you have any active resources in your account, do the following:
Open the Billing and Cost Management console.
Choose Bills in the navigation pane.
You can see the charges incurred by different services in the Bill details by service section.
You can see the charges incurred in different AWS Regions in the Bill details by account section.
For each service, identify the Regions where the services have incurred charges.
To terminate the identified active resources under different services, do the following:
Open the AWS Management Console.
For Find services, enter the service name.
After opening the service console, terminate all your active resources. Be sure to check each Region where you have allocated resources.
if the issue is a corrupted terraform state, perhaps storing the state in a versioned S3 bucket would help reduce the impact of that.
Use Terraformer to import all resources into terraform configuration then do whatever you want:
terraformer import aws --resources="*"
https://github.com/GoogleCloudPlatform/terraformer
Take care of your state file lock f.e. by using dynamodb & enable s3 versioning.
Is there an easy way to nuke all resources in an AWS account
belonging to a particular owner?
Since you are using Terraform, you can use Blank Apply.
It will destroy all the Resources in a state file.
We use terraform to set this up but sometimes due to corruption, the
state becomes inconsistent.
It's better to use version control system to avoid drifts and inconsistencies in your state file and use remote states in order to make sure everyone is on the same page.
Lets say an AWS stack was created using CloudFormation.
Now one of those resources was modified outside CloudFormation.
1) Is it possible to have CloudFormation specifically create those resources? Based on my understanding, we can't do that because CloudFormation does not identify a difference, and so does not create the modified resources. Is my observation correct?
2) Also, what options do I have to revert a stack to its original state, if modified outside CloudFormation?
This is one possible hack you could use without deleting the entire stack.
From the template remove the specific resource which got deleted accidentally.
Now update the stack which makes your stack and resources in your account in sync.
Revert the template to its state before step1 and update again which will create the resource which got deleted accidentally.
Unfortunately the answer for both your questions is NO.
If you modify the resources in the stack after stack creation status is COMPLETE, there is nothing CF can do since it doesn't keep track of modification to resources
You have no option other than deleting the current stack and create a new one
First, beware that modifying CloudFormation-created resources outside of CloudFormation is explicitly discouraged, according to AWS CloudFormation Best Practices:
Manage All Stack Resources Through AWS CloudFormation
After you launch a stack, use the AWS CloudFormation console, API, or AWS CLI to update resources in your stack. Do not make changes to stack resources outside of AWS CloudFormation. Doing so can create a mismatch between your stack's template and the current state of your stack resources, which can cause errors if you update or delete the stack.
However, if you've modified a CloudFormation-managed resource accidentally and need to recover, you may have some limited options beyond simply deleting and re-creating the stack altogether (which may not be an acceptable option):
It is not possible for CloudFormation to automatically update its internal state based on the current state of an externally-modified resource.
However, depending on the exact resource type, in some cases you can manually update CloudFormation afterwards by applying a stack update that matches the current state of the resource.
Similarly, it is not possible for CloudFormation to automatically revert an externally-modified resource back to its original unmodified CloudFormation state.
However, depending on the exact resource type, in some cases you can either:
Revert a resource by manually updating the resource back to its original state;
Update the resource by applying a stack update, bringing both the CloudFormation stack and the managed resource to an altogether new state that will once again be in sync.
To force the EC2 re-creating, I do use a simple trick, when I'm deploying, I jump between AMI's IDs (I took two similar AMI's ID), that had helped me when I'm testing user data or things that I want to test during the EC2 bootstrap. Again, it just works for EC2.
Unfortunately, the answer is NO
if you made changes in the stack after the creation, Cloudformation can't track those changes.
if you need to revert those changes, you must delete the stack and rebuild.