Lets say an AWS stack was created using CloudFormation.
Now one of those resources was modified outside CloudFormation.
1) Is it possible to have CloudFormation specifically create those resources? Based on my understanding, we can't do that because CloudFormation does not identify a difference, and so does not create the modified resources. Is my observation correct?
2) Also, what options do I have to revert a stack to its original state, if modified outside CloudFormation?
This is one possible hack you could use without deleting the entire stack.
From the template remove the specific resource which got deleted accidentally.
Now update the stack which makes your stack and resources in your account in sync.
Revert the template to its state before step1 and update again which will create the resource which got deleted accidentally.
Unfortunately the answer for both your questions is NO.
If you modify the resources in the stack after stack creation status is COMPLETE, there is nothing CF can do since it doesn't keep track of modification to resources
You have no option other than deleting the current stack and create a new one
First, beware that modifying CloudFormation-created resources outside of CloudFormation is explicitly discouraged, according to AWS CloudFormation Best Practices:
Manage All Stack Resources Through AWS CloudFormation
After you launch a stack, use the AWS CloudFormation console, API, or AWS CLI to update resources in your stack. Do not make changes to stack resources outside of AWS CloudFormation. Doing so can create a mismatch between your stack's template and the current state of your stack resources, which can cause errors if you update or delete the stack.
However, if you've modified a CloudFormation-managed resource accidentally and need to recover, you may have some limited options beyond simply deleting and re-creating the stack altogether (which may not be an acceptable option):
It is not possible for CloudFormation to automatically update its internal state based on the current state of an externally-modified resource.
However, depending on the exact resource type, in some cases you can manually update CloudFormation afterwards by applying a stack update that matches the current state of the resource.
Similarly, it is not possible for CloudFormation to automatically revert an externally-modified resource back to its original unmodified CloudFormation state.
However, depending on the exact resource type, in some cases you can either:
Revert a resource by manually updating the resource back to its original state;
Update the resource by applying a stack update, bringing both the CloudFormation stack and the managed resource to an altogether new state that will once again be in sync.
To force the EC2 re-creating, I do use a simple trick, when I'm deploying, I jump between AMI's IDs (I took two similar AMI's ID), that had helped me when I'm testing user data or things that I want to test during the EC2 bootstrap. Again, it just works for EC2.
Unfortunately, the answer is NO
if you made changes in the stack after the creation, Cloudformation can't track those changes.
if you need to revert those changes, you must delete the stack and rebuild.
Related
Our team had an issue where someone manually modified an IAM role for an operational event. We hoped that a CloudFormation stack redeployment would revert the state of the IAM role, however, the manual change was still there.
My working theory is that since the IAM role arn is the same, that CloudFormation does not delete and recreate it. Is that accurate? And if so, how do we ensure all relevant resources are torn down during a deployment?
My working theory is that since the IAM role arn is the same, that CloudFormation does not delete and recreate it. Is that accurate?
CloudFormation (CFN) does not check for any changes made outside of its control. You could remove the role, and CFN would still "think" that the role is there.
If you change a resource created by CFN manually outside of CFN (bad practice), you have so called a stack drift. CFN by itself is not aware of any changes made to resources it creates that occurred outside its control. But, CFN provides special tools which you have to explicitly call to detect the drift:
Detect drift on an entire CloudFormation stack
Not all resources support drift detection, but AWS::IAM::Role is one which does.
And if so, how do we ensure all relevant resources are torn down during a deployment?
Not sure what do you mean here. But you have to manually fix the drift. You have four choices:
Change the role back to its original state,
Update template to reflect the external changes,
Use import to import the changed role to stack,
Delete the entire stack, and create new one from the original template.
The last choice ensures that the modified role is also deleted and recreated in its original form.
I am having an interesting problem with AWS. The project I'm working on create's set of various AWS service instances in one predetermined zone. For example, one VPC, one firewall, one S3 bucket, RDS instance..
Now if one of these fails to be created for the lack of the general available resources in the region I have to go through nasty deletion and switch to another region.
Is there something in AWS API that can tell if there is enough resources to create instance of something without actually creating it?
if one of these fails to be created for the lack of the general available resources in the region I have to go through nasty deletion and switch to another region
Not a direct answer to your question (and I would have put it as a comment, but I lack reputation to comment... hint hint), but if you were to use CloudFormation, one of the very nice features of it is that if any one resource fails to create, it will roll back the entire stack. The other nice thing about CloudFormation is it will give you consistent infrastructure across your multiple regions. Have you explores using CloudFormation to deploy your resources.
I'm trying to do the following: the parent Stacks launches the first child Stack which creates a fully configured EC2 instance. Once that is completed, the parent Stack kicks off a second Stack that uses a Lambda function to create an AMI, which is then used for an AutoScaling setup even further downstream. This is working perfectly.
Now the challenge: when I update the metadata for the EC2 instance from the first child Stack I would really like the second Stack to be triggered. In other words: I want to be able to change the seed instance and have the CloudFormation Stack update, creating a new AMI.
I'm able to get the seed instance to update, but the second child Stack isn't triggered :-(
I've Google everything I could think of, but Update Policy doesn't apply, manually kicking off the second child is defying the point of having nested Stacks and I'm pretty sure I'm missing some obvious feature or clever trick, so I'm asking you guys to help me out. Please.
Have you tried using Lambda-backed custom resource? You can have the service token of resource as the Lambda and use DependsOn with the first nested stack. It will kick off whenever the CF script runs or updates.
You can also lookup the stack itself from the Lambda function to determine any changes if you want.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
Lets imagine that i have a cloudformation stack with 2 EC2 instances 1 S3 bucket and 1 dynamoDb table.
Everything is good with this stack until we decide to do some refactoring and split this stack into two stacks. Stack A that has only 2 EC2 instances and Stack B that has 1 S3 bucket and 1 dynamoDb table.
My question is: what is the best way to do that without destroying and re creating the resources or doing migration? (i am looking for a solution similar to terraform move state).
Could you please share how you organise your stacks and how you refactor them.
Moving resources between stacks is now supported for some resources including DynamoDb, EC2 instances and S3 buckets. The aws docs at
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/refactor-stacks.html give the full detail and instructions starting with:
Using the resource import feature, you can move resources between, or refactor, stacks. You need to first add a Retain deletion policy to the resource you want to move to ensure that the resource is preserved when you remove it from the source stack and import it to the target stack.
Important
Not all resources support import operations. See Resources that
Support Import Operations before you remove a resource from your
stack. If you remove a resource that doesn't support import operations
from your stack, you can't import the resource into another stack or
bring it back into the source stack.
Currently, there is no mechanism in CloudFormation to move objects/resources from 1 stack to another.
If you had used Terraform to deploy your resources, then yes, you could use terraform state mv to move resources from one state file to a different state file. A Terraform state file is associated with a single Terraform deployment, so you you could think of it as being analogous to how CloudFormation stores its version of what's deployed in a stack.
Note that Terraform state moves are not the simplest things in the world and AFAIK there is no GUI interface to assist with moves. Make backups and set aside time for testing.
There is, however, a project called Terraboard that is a GUI that can help you inspect the state file.
How can I force delete my AWS cloud formation stack? The tied resources were already deleted previously like the DNS, route53, etc. Whenever I delete the stack, it failed because it couldn't find the resources.
The specified hosted zone does not exist. DELETE_FAILED.
This made the deletion halt and made the stack still active. How can I force clean-up and remove this stack?
Based on this thread, it appears that stack deletion can be problematic if the stack is altered.
https://forums.aws.amazon.com/thread.jspa?threadID=71050&start=0&tstart=0
Looks like you need to go via Support to get it deleted.
That said, I have been able to delete stacks where I have manually altered the components. Sometimes you need to wait a couple of hours after you terminate instances, VPCs, subnets etc before the stack deletion actually succeeds.
AWS Cloud formation stack does not get deleted until all dependency resources either used in other stack or busy with non stack resources. So better see which resource is shared some where else and try to release that.
hope that help.
It seems the Route53 records were either deleted manually before cloudformation or there was some failure within cloudformation after it deleted the record. Its stuck because it can't find the hosted zone because it was already deleted if indeed this is what was described in your post. You can try to update the stack with the original script and then delete the stack via cloudformation without any manual intervention.
Check your IAM role and the policy, review the privileges that you have associated with the role, so that if there is any permission issue, that should resolve the issue.
Refer: https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-stack-delete-failed/
Need to check the reason for failure in stack details
Either delete this manually, or retain those resources while eleteing
eg:
$ aws cloudformation delete-stack --stack-name my-stack --retain-resources myresource1 myresource2