I am using removalPolicy: cdk.RemovalPolicy.DESTROY.
The other two options are RETAIN and SNAPSHOT.
If I delete my table from the console and try to create using the cdk it gives an error say could not find resource.
Question -- what option I can use if the script is unable to find the table then it should create ?
The RemovalPolicy has nothing to do with things you remove yourself outside of CDK. This is a bad idea (and in some cases not permitted) as you are supposed to delete resources by updating your CDK code to remove that resource, and then redeploying it.
The RemovalPolicy tells CloudFormation what to do if you change your CDK code so that the resource is no longer part of your CDK stack.
For example if you have an S3 bucket you cannot rename it, but you can still change its name in your CDK stack. If you do, CDK will need to remove the old S3 bucket and create a new one with the new name. The same applies for a number of other resources that can't be renamed, such as DynamoDB tables.
How CDK handles removing this old resource is what the RemovalPolicy is for. If you set it to RETAIN it will just forget about it and leave it up to you to clean up manually later. Using the DESTROY policy tells CDK to try to delete your resource automatically along with all the data it contains.
Usually you would use DESTROY if the data is not important and can be easily recreated (e.g. cache data), and RETAIN if the data is important and you would not want to lose it (e.g. user data).
Often it is a good idea just to always use RETAIN. This way if you accidentally make a typo in your CDK stack and rename a resource by mistake, all the data in it won't get deleted!
what option I can use if the script is unable to find the table then it should create ?
You just create the resource normally. When you write a CDK stack, you are not telling it what to do (create this S3 bucket, create this DynamoDB table) but rather you are telling it what you want (I want an S3 bucket with this name, and a DynamoDB table with that name). CDK will figure out which resources need to be created to meet your request, and if CDK has already created those resources in an earlier deploy, it will just update them if changes are needed or leave them untouched if no changes are required.
The reason you got an error after you deleted the resource manually is because CDK was trying to find it to figure out whether it needed to be updated or not. This is why you should never change any AWS resources manually that were configured by CDK - always update the CDK template and redeploy. If you fiddle with the resources manually it is very easy to break CDK, and in that situation the only solution is to destroy the stack, manually clean up any resources that couldn't be destroyed, then redeploy it from scratch (and then also reupload any user data you might have, often a big deal!)
CDK RemovalPolicy is equivalent to Cloudformatoin DeletePolicy which takes into effect when a resource is removed from CDK/Cloudformation.
DESTROY: This is default option, removes actual resource if code is removed in CDK.
RETAIN: This will retain the actual resource, if resource code is removed from CDK.
SNAPSHOT: This will also deletes the resource if resource code is removed from CDK but will create SNAPSHOT before deleting. Ex: RDS cluster or EC2 Volume.
These options are applicable when actual resource is removed from CDK code, but not from AWS. if a resource created by CDK/Cloudformation is manually deleted, it can no longer be maintained by CDK and will result in errors unless id is changed ex: MyQueue is changed to MyQueueSomething. This will result in creation of new queue and removal of old queue. since old queue is not existing it will be ignored.
new sqs.Queue(this, 'MyQueue', {
encryption: sqs.QueueEncryption.KMS_MANAGED
});
If we mistakenly delete a resource manually outside cdk/cf and we want to continue managing it via cdk/cf, we have to manually create the resource with same physical id. Here are some more details.
Related
Goal:
I need to create an AWS ManagedPolicy that contains ALLOW permissions for API actions on resources created in a pre existing stack. No I cannot modify the existing stack template and simply add a policy to it. I need to create a new stack that deploys a policy that enables actions on the existing stacks resources
Solution:
Create a CDK project to generate and deploy this policy stack. Within this CDK project I want to load the existing stack and iterate over its resources adding permissions to my new stack's policy.
Problem:
I don't see any way to load an existing stack in CDK. I was hunting around for a "Stack.fromArn(...)" but don't see anything even similar.
Question:
Is there some obsucre way to do this? Or is it simply not supported?
I did not tried it, however it looks like if you can access/lookup at least one construct from the existing stack, you can use the method Stack.of(construct) https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_core.Stack.html#static-ofconstruct to lookup the first stack scope in which the construct is defined. Not sure however how you could iterate resources in the looked up stack construct.
It might be not be the best answer, however one option could be to export the outputs for resources in existing stack which you want to include in the policy, and import these values in the new stack where you create the policy.
Using template.yaml, Cloudformation service created stack having three s3 buckets.
I deleted one s3 resource manually.
Say the stack name is stack1
On running the same template.yaml again(unchanged), with stack name stack1, following this update_procedure
does CloudFormation service update the same stack? with that
missing bucket....it is not updating with missing s3 bucket in my case
You can't create two stacks with the same name in the same region. If you were to do this in another region it would create the bucket you deleted but fail to create the other buckets, all assuming you named your buckets in the template. If the buckets we not named (so CloudFormation created the names for you) then it will create all three buckets, but the names will not be the same as they were before.
CloudFormation will not update a stack when you tell it to create a stack.
EDIT:
Based on your updated question, it seems you are asking if the bucket will be recreated. The answer to that is no. CloudFormation sees that nothing has changed in what you've asked for, so no action is taken. As a matter of fact you should get an error when updating, saying something along the lines of "no changes".
There are exceptions to the above "no", but for your purposes here I think it's sufficient.
The easiest solution for you is to remove the S3 bucket, that you deleted, from the template, run the update (it will "delete" it even though it's already gone) and then add it back to the template and update it again. That will cause it to be created again.
If you are worried about this sort of thing happening in the future consider using Drift Detection with CloudFormation.
I am trying to add a 'get' function to an already existing DynamoDB table in AWS. I added a yml file for the table, and when I tried to deploy the stack, it said that the resource for my table already exists.
In the past I have deleted the table when this happens and then redeployed the stack, but in this case I do not want to delete the table and don't know how else to fix the problem.
My first issue was that I had defined the table's schema incorrectly in the YML file. Then when I redefined it I had the issue with the resource already existing in the stack.
My error:
Serverless Error ---------------------------------------
An error occurred: ConsumersTable - dev-con already exists in stack arn:aws:cloudformation:us-ea
I experienced this issue myself and contacted AWS support. It seems that CloudFormation associates the logical ID with the resource, for example (using IAM as an example but many resources are the same):
exampleLogicalId:
Type: AWS::IAM::ManagedPolicy
Properties:
PolicyDocument:
ManagedPolicyName: exampleName
...
Here, the resource gets created with a custom name (exampleName in this case). As long as the logical ID (exampleLogicalId) remains unchanged, CloudFormation will recognise that the resource has already been created and will update it as necessary.
However if you change the logical ID (as happened to me when using CDK, because it auto-generates these logical IDs) then CloudFormation thinks it needs to create a new resource. But because the name already exists, and for this resource type the names must be unique, there is a conflict and the creation fails. This seems to be because CloudFormation creates all new resources before deleting any removed ones.
The solution is to either 1) put the logical ID back to what it originally was so CloudFormation recognises it as an 'update' rather than a 'create', or 2) change the unique part (the exampleName in this example) to be something unique.
Changing the name (or whatever the unique field is for the resource in question) can be a convenient option, because it will create the new resource and associate it with the new logical ID, then delete the old resource if that logical ID no longer exists in your CloudFormation template. You can then rename the resource back to what you wanted originally (keeping the same logical ID) and deploy a second time, and CloudFormation will recognise it as an update operation, and rename the resource back to what you wanted the first time.
Just be aware that doing this will not perform an update, but rather a delete-then-create. So if your resource has data (e.g. DynamoDB table with data, IAM policy already attached to roles, Parameter Store entry with a value entered) you may need to recreate this data for the new resource.
You can import existing resources to CloudFormation (CFN):
Bringing Existing Resources Into CloudFormation Management
AWS::DynamoDB::Table is also one of the resources that supports the import operation.
Thus, if you don't want to delete your existing table, you can import it into CFN. Then you can manage it using CFN as would any other table created from scratch in CFN.
Hopes this helps.
My use case is that we already have a stack created out of AWS Cloudformation.
Now I want to update that stack and my requirement is that I want to delete a resource that was already created and add the new modified resource but I want to make sure that the delete happens before the create part.
I explored the dependsOn but that helps me with setting the order of resource creation. It doesn't help with ensuring the delete and the create ordering (or atleast nothing that i could find)
How to make sure that the resource deletion happens before resource creation while doing the cloudformation update
I understand you want to,
delete a resource that was already created and add the new modified
resource
Below is my understanding, let me know if it helps,
It is very trickey to Delete and Create resource having same resource name/dependency in a single CloudFormation deployment.
Easiest approach :
First deploy CFN template to Delete a resource i.e. remove the code
from template and than add new resource/modified one. While doing
that you need to check if "retention policy" is in place because if
you are retaining deleted resources than CloudFormation will not
create same resource again.
Than deploy the CFN template to create/modify resources
Other approach might be:
If you want to ensure the resource deletes before creating new one,
in a single template, you might need to create a nested stack
for resource deletion and resource creation
And add dependency on the deletion cloudFormation template i.e.
Create resource template will depends on Delete resource template.
There are also AWS::CloudFormation::WaitCondition which can be
used here.
Also I think anyway you will receive an error if you try to create/modify on deleted/ delete in-progress resource
There is no other option unless dividing your operation into two steps:
You will need to update the stack, using the current template, changing only the name of the resources which you will modify later.
Update the stack - uploading a new template with your modifications - but remember to set the resource name with the previous value.
I have a cloudformation template that creates an S3 bucket as part of a cloudformation stack. On the new version of my template, I 'm planning to migrate my application from S3 to EFS.
Is there a way to remove the S3 bucket resource from the template, without having it deleted? Ideally, I would like my older users to have the s3 bucket available after they upgrade, but for the new users to not have it at all. It looks like DeletionPolicies could help here, but the documentation on it says that it only applies to stack deletion, but not upgrades.
Going to elaborate on user3470009's answer.
The main, advertised purpose of the DeletionPolicy is to keep a resource when a stack is deleted. It's mentioned almost as an afterthought in the AWS docs for DeletionPolicy that it also functions during resource removal from a stack:
Note that this capability also applies to stack update operations that
lead to resources being deleted from stacks. For example, if you
remove the resource from the stack template, and then update the stack
with the template.
So the workflow to remove a resource from a stack without deleting the actual resource is:
Add "DeletionPolicy" : "Retain" to the resource declaration in your CF template
Apply changes by either saving in the UI or running aws cloudformation on the CLI or whatever other tool you use
Check in the UI that your resource has the correct changes. There are some gotchas about when CF doesn't update the metadata. See the docs link above
Remove the resource from your template
Apply changes. Watch the events log to see that it says DELETE_SKIPPED:
2018-10-15T15:32:32.956Z HostedZone DELETE_SKIPPED
Setting a DeletionPolicy of "Retain" will cause the bucket itself to remain after a stack update that deletes the resource.
I came across this question requiring a slight variation. I needed to extract my bucket to another stack and can not delete it in the move. This method worked well:
create a new stack with the bucket in question. (note: you now have 2 stacks referencing the same bucket)
remove the bucket from the original stack. The resource is deleted from the original stack but not from S3 since it is still referenced in your new stack.
I also tested Houser's response above and confirmed the bucket will not be deleted if it contains files. While this works, it does attempt to delete the bucket 3 times before it completes (and reports errors each time). migrating to a new stack will not throw any errors.
When you remove a resource from your template, and update a stack from this template, the resources will be deleted. There is no way to avoid that.
Since your existing users will continue using the S3 bucket, I would recommend preserving the bucket in your template. Remove it when the bucket has been removed from your product completely.
If needed, you could version your template (old vs. new).
If you absolutely need to remove the bucket from your template, you may be able to use a loophole. When CloudFormation deletes a bucket, the bucket must be empty. If it's not empty, then the bucket should be preserved and removed from your stack. You could experiment and see if it works for you. If it works in testing, then you can try using it in production.