Our team had an issue where someone manually modified an IAM role for an operational event. We hoped that a CloudFormation stack redeployment would revert the state of the IAM role, however, the manual change was still there.
My working theory is that since the IAM role arn is the same, that CloudFormation does not delete and recreate it. Is that accurate? And if so, how do we ensure all relevant resources are torn down during a deployment?
My working theory is that since the IAM role arn is the same, that CloudFormation does not delete and recreate it. Is that accurate?
CloudFormation (CFN) does not check for any changes made outside of its control. You could remove the role, and CFN would still "think" that the role is there.
If you change a resource created by CFN manually outside of CFN (bad practice), you have so called a stack drift. CFN by itself is not aware of any changes made to resources it creates that occurred outside its control. But, CFN provides special tools which you have to explicitly call to detect the drift:
Detect drift on an entire CloudFormation stack
Not all resources support drift detection, but AWS::IAM::Role is one which does.
And if so, how do we ensure all relevant resources are torn down during a deployment?
Not sure what do you mean here. But you have to manually fix the drift. You have four choices:
Change the role back to its original state,
Update template to reflect the external changes,
Use import to import the changed role to stack,
Delete the entire stack, and create new one from the original template.
The last choice ensures that the modified role is also deleted and recreated in its original form.
Related
At my company, our CI/CD pipleline can only connect to AWS via an assumed IAM role. I'm trying to lock down this role so that it can only update resources that are part of the stack, to avoid mistakenly changing a resource that is managed by another team.
I am running into a bit of a chicken or the egg scenario here. It seems that in order to create a role that restricts access to just the resources created by my stack, I need the IDs of those resources, which means the stack needs to have been created. I could run the stack with admin permissions once, then extract the resource IDs, and then put it into CI/CD, but that seems to defeat the purpose.
My ideal solution would be to be able to create a set of permissions that allow CloudFormation to create a fixed set of Resource Types, and then grant additional permissions to only those resources that were created as part of the stack. Is this possible? Or is there a different philosophy for how to manage the permissions for CI/CD?
What has me confused is how you want to "lock down" the resource to a particular stack. Anyway, try this:
Have a role that only have Create* permissions (You prob need Describe* too). This role is used to create new stacks and so it's not possible for it to meddle with resources already created.
In your CloudFormation template, create an IAM role that has Update* / Delete* permission to your resource. Call into this role when you want to modify the stack. Maybe give it a name of !Sub ${AWS::StackName}-CfnRole.
You'll also need to give it a deletion policy of retain and delete it manually after the stack is cleaned up.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html. Otherwise, when you call that role to delete a stack, you'll run into permission denied error as the role will almost certainly be the first resource to be deleted.
?? Does that work for you
I have autoscaling configured in one of my dynamoDB table which is deployed through cloudformation. Once i have disabled the autoscaling though the console. Now when i deploy the autoscaling through cloudformation again, its disabled though it should be enabled. I have already tried update policy but doesn't seems to help.
UpdatePolicy:
AutoScalingReplacingUpdate:
WillReplace: Boolean
Is the cloudformation not smart enough to override the manual changes? And is there any specific way to handle this.
I noticed that the drift status does not capture the manual changes.
The UpdatePolicy attribute doesn't support the AWS::DynamoDB::Table resource type.
According to the docs:
Use the UpdatePolicy attribute to specify how AWS CloudFormation handles updates to the AWS::AutoScaling::AutoScalingGroup, AWS::Lambda::Alias, or AWS::ElastiCache::ReplicationGroup resources.
Note that for drift detection to pick up on changes, you'll need to make sure that all properties in your cloudformation resource are explicitly set (including default values) in order for it to properly detect drift.
See in the docs
CloudFormation only determines drift for property values that are explicitly set, either through the stack template or by specifying template parameters. This does not include default values for resource properties. To have CloudFormation track a resource property for purposes of determining drift, explicitly set the property value, even if you are setting it to the default value.
Hope this helps!
Lets imagine that i have a cloudformation stack with 2 EC2 instances 1 S3 bucket and 1 dynamoDb table.
Everything is good with this stack until we decide to do some refactoring and split this stack into two stacks. Stack A that has only 2 EC2 instances and Stack B that has 1 S3 bucket and 1 dynamoDb table.
My question is: what is the best way to do that without destroying and re creating the resources or doing migration? (i am looking for a solution similar to terraform move state).
Could you please share how you organise your stacks and how you refactor them.
Moving resources between stacks is now supported for some resources including DynamoDb, EC2 instances and S3 buckets. The aws docs at
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/refactor-stacks.html give the full detail and instructions starting with:
Using the resource import feature, you can move resources between, or refactor, stacks. You need to first add a Retain deletion policy to the resource you want to move to ensure that the resource is preserved when you remove it from the source stack and import it to the target stack.
Important
Not all resources support import operations. See Resources that
Support Import Operations before you remove a resource from your
stack. If you remove a resource that doesn't support import operations
from your stack, you can't import the resource into another stack or
bring it back into the source stack.
Currently, there is no mechanism in CloudFormation to move objects/resources from 1 stack to another.
If you had used Terraform to deploy your resources, then yes, you could use terraform state mv to move resources from one state file to a different state file. A Terraform state file is associated with a single Terraform deployment, so you you could think of it as being analogous to how CloudFormation stores its version of what's deployed in a stack.
Note that Terraform state moves are not the simplest things in the world and AFAIK there is no GUI interface to assist with moves. Make backups and set aside time for testing.
There is, however, a project called Terraboard that is a GUI that can help you inspect the state file.
Lets say an AWS stack was created using CloudFormation.
Now one of those resources was modified outside CloudFormation.
1) Is it possible to have CloudFormation specifically create those resources? Based on my understanding, we can't do that because CloudFormation does not identify a difference, and so does not create the modified resources. Is my observation correct?
2) Also, what options do I have to revert a stack to its original state, if modified outside CloudFormation?
This is one possible hack you could use without deleting the entire stack.
From the template remove the specific resource which got deleted accidentally.
Now update the stack which makes your stack and resources in your account in sync.
Revert the template to its state before step1 and update again which will create the resource which got deleted accidentally.
Unfortunately the answer for both your questions is NO.
If you modify the resources in the stack after stack creation status is COMPLETE, there is nothing CF can do since it doesn't keep track of modification to resources
You have no option other than deleting the current stack and create a new one
First, beware that modifying CloudFormation-created resources outside of CloudFormation is explicitly discouraged, according to AWS CloudFormation Best Practices:
Manage All Stack Resources Through AWS CloudFormation
After you launch a stack, use the AWS CloudFormation console, API, or AWS CLI to update resources in your stack. Do not make changes to stack resources outside of AWS CloudFormation. Doing so can create a mismatch between your stack's template and the current state of your stack resources, which can cause errors if you update or delete the stack.
However, if you've modified a CloudFormation-managed resource accidentally and need to recover, you may have some limited options beyond simply deleting and re-creating the stack altogether (which may not be an acceptable option):
It is not possible for CloudFormation to automatically update its internal state based on the current state of an externally-modified resource.
However, depending on the exact resource type, in some cases you can manually update CloudFormation afterwards by applying a stack update that matches the current state of the resource.
Similarly, it is not possible for CloudFormation to automatically revert an externally-modified resource back to its original unmodified CloudFormation state.
However, depending on the exact resource type, in some cases you can either:
Revert a resource by manually updating the resource back to its original state;
Update the resource by applying a stack update, bringing both the CloudFormation stack and the managed resource to an altogether new state that will once again be in sync.
To force the EC2 re-creating, I do use a simple trick, when I'm deploying, I jump between AMI's IDs (I took two similar AMI's ID), that had helped me when I'm testing user data or things that I want to test during the EC2 bootstrap. Again, it just works for EC2.
Unfortunately, the answer is NO
if you made changes in the stack after the creation, Cloudformation can't track those changes.
if you need to revert those changes, you must delete the stack and rebuild.
How can I force delete my AWS cloud formation stack? The tied resources were already deleted previously like the DNS, route53, etc. Whenever I delete the stack, it failed because it couldn't find the resources.
The specified hosted zone does not exist. DELETE_FAILED.
This made the deletion halt and made the stack still active. How can I force clean-up and remove this stack?
Based on this thread, it appears that stack deletion can be problematic if the stack is altered.
https://forums.aws.amazon.com/thread.jspa?threadID=71050&start=0&tstart=0
Looks like you need to go via Support to get it deleted.
That said, I have been able to delete stacks where I have manually altered the components. Sometimes you need to wait a couple of hours after you terminate instances, VPCs, subnets etc before the stack deletion actually succeeds.
AWS Cloud formation stack does not get deleted until all dependency resources either used in other stack or busy with non stack resources. So better see which resource is shared some where else and try to release that.
hope that help.
It seems the Route53 records were either deleted manually before cloudformation or there was some failure within cloudformation after it deleted the record. Its stuck because it can't find the hosted zone because it was already deleted if indeed this is what was described in your post. You can try to update the stack with the original script and then delete the stack via cloudformation without any manual intervention.
Check your IAM role and the policy, review the privileges that you have associated with the role, so that if there is any permission issue, that should resolve the issue.
Refer: https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-stack-delete-failed/
Need to check the reason for failure in stack details
Either delete this manually, or retain those resources while eleteing
eg:
$ aws cloudformation delete-stack --stack-name my-stack --retain-resources myresource1 myresource2