I am using aws-cdk and $cdk deploy to deploy some stacks.
However, there comes error like this
21:12:30 | CREATE_FAILED | AWS::S3::Bucket | S3BucketStaticResourceB341FA19
si2-s3-sbu-mytest-xxx-static-resource-5133297d-91 already exists
Normally this kind of error, I can find the item exists in AWS console.
However in this case, $aws s3 ls doesn't show the bucket named this.
Why does this occur or where should I fix??
Generally, it is a good idea to avoid explicitly providing physical names for resources you create with CDK.
The documentation explains the reasoning:
Assigning physical names to resources has some disadvantages in AWS CloudFormation. Most importantly, any changes to deployed resources that require a resource replacement, such as changes to a resource's properties that are immutable after creation, will fail if a resource has a physical name assigned. If you end up in that state, the only solution is to delete the AWS CloudFormation stack, then deploy the AWS CDK app again. See the AWS CloudFormation documentation for details.
So if you introduce a change that requires your bucket to be replaced, you'll see the aforementioned error.
In your specific case, it is probably an S3-specific issue that bucket names are globally unique - across all accounts and regions, as stated by #Omar Rosadio in the comments. This makes naming your buckets yourself an especially bad idea.
If you don't pass the bucketName property when creating the bucket, CDK will generate a unique name for you, so you don't have to worry about this, and I suggest doing so.
Related
I created and deployed a S3 resource (bucket) using cloudformation.
After that i deployed a version without that resource.
Then i deployed a version with the resource.
Since the bucket exists, it gives me an error that it cannot deploy.
This has happened to me before, in past times I deleted the resource and deployed again.
I'm looking for a way to use the resource for future deployments. It is the exact same resource, this is the yaml :
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub "myBucketName"
Is there anything I can add to the resource, a policy, a unique ID, anything so that i could use the existing resource?
Thanks!
To "use the existing resource" in a CFN, you have to import it. Also its a bad practice to keep modify resources created by CFN outside of CFN. This leads to drift and number of issues, one of which you are experiencing.
IAM resources are global, meaning they aren't isolated within specific AWS regions. However, the documentation for an IAM role includes a warning:
Important
Naming an IAM resource can cause an unrecoverable error if you reuse the same template in multiple regions. To prevent this, we recommend using FN::Join and AWS::Region to create a region-specific name, as in the following example ...
What kind of "unrecoverable error" are they talking about? Will cloudformation just fail to create the resource, or will things get stuck in some weird state?
The stacks we have which include IAM resources only contain IAM resources, so I suspect I may be able to ignore this warning.
The problem they want you to be aware of is that IAM is a global namespace, which can result in problems, if you don't manually namespace resources.
Here's an example:
Stack 1 in eu-central-1 creates a role with the name AppAdmin
Stack 2 in eu-west-1 creates a role with the name AppAdmin - this stack will fail to create or update
This failure is usually nothing life-threatening, it just means, your deployment will be broken.
If it's an update of an existing stack, a rollback will be performed.
If it's a new stack, the stack will fail to create and you need to manually delete the stack afterwards before rolling it out again (by default the orphaned resources will be deleted).
You can simply avoid this problem by namespacing the resources as suggested:
Stack 1 in eu-central-1 creates a role with the name AppAdmin-eu-central-1
Stack 2 in eu-west-1 creates a role with the name AppAdmin-eu-west-1
Now everybody is happy!
(The same thing is true for S3 Bucket names and other resources with a global namespace)
How can I get the s3 bucket name from the bootstrap stack?
I cant seems to find any other method than creating a dummy asset and get the s3BucketName attribute.
--
Update:
I have two stacks:
Stack 1: Creating a bunch of S3 assets.
Stack 2: Creating an IAM policy to give access to the assests created in Stack 1 (no assets)
I need to lookup the bootstrapped assets bucket in Stack 2 to be able to add the resources attribute in my IAM Policy
I find multiple solutions for this, but all of them seems hacky
The CDK framework can reference the bucket when instantiating new assets, without me providing the bucket name, this implies that the framework know the bucket name, hence it must be possible for me to get hold of it and avoid either hardcoding, creating dummy assets or using Fn::Import when adding the resources attribute in my IAM Policy in Stack 2
To answer my own question, you can get the asset bucket name through the DefaultStackSynthesizer:
example:
const assetBucketName = cdk.DefaultStackSynthesizer.DEFAULT_FILE_ASSETS_BUCKET_NAME
Toolkit reference
cdk bootstrap --toolkit-bucket-name <bucket name>
I'm finding that cdk tries to recreate S3 buckets every time I deploy. If I don't specify a bucket name, it generates a new junk bucket name every time. If I do specify a name, it refuses to deploy because the bucket already exists. How can I make it "upsert" a bucket?
Here's the code I'm using:
const dataIngestBucket = new Bucket(this, 'data-lake', {
bucketName: `${this.props.environmentName}-my-company-data-lake`
});
As long as I do not see the language you want to use, I give an answer using python. It can be easily traced and converted to any other languages.
Please refer to aws_cdk.aws_s3.Bucket class.
There you will find parameters to specify during class creating which allow you reach your goal, namely auto_delete_objects=True and removal_policy=cdk.RemovalPolicy.DESTROY.
CDK would do an update on your stack resources automatically if CDK code is updated.
For example, when you execute a CDK stack that creates a bucket for the first time, bucket would be created with provided configuration.
When you update your CDK code to say update lifecycle policy of the bucket or add a CORS, as part of the same stack, the update of the stack would automatically update the bucket - it would not recreate the bucket as Cloud Formation knows that there is an update on an existing stack.
In your case, it seems the stack is being re-created after a removal when the stack resources still exists. This causes Cloud Formation to create a new stack and its resources which were not removed when stack was destroyed.
Generally, issues occur when stack update fails and it is in rollback state, for example. In that case, redeploy would try to create the bucket again and fail.
In that case, possible option could be:
Delete the buckets
Delete the stack
Redeploy to re-create
Many times, we do not want to delete the resources, as they contain data; in that case, you can use another library such as boto3 for python in the CDK code, to check if the resource exists - if not create via CDK. This would cause CDK code to be not attempt creating the bucket if it exists ( CDK itself cannot be used to see if say S3 resource exists already - at least have not seen how to achieve this)
Another important point is the removal policy associated with the resource
troubleshooting_resource_not_deleted
My S3 bucket, DynamoDB table, or other resource is not deleted when I
issue cdk destroy
By default, resources that can contain user data have a removalPolicy
(Python: removal_policy) property of RETAIN, and the resource is not
deleted when the stack is destroyed. Instead, the resource is orphaned
from the stack. You must then delete the resource manually after the
stack is destroyed. Until you do, redeploying the stack fails, because
the name of the new resource being created during deployment conflicts
with the name of the orphaned resource.
If you set a resource's removal policy to DESTROY, that resource will
be deleted when the stack is destroyed.
However, even with the removal policy as DESTROY, Cloud formation cannot delete a non-empty bucket. Extract from the same link below -
AWS CloudFormation cannot delete a non-empty Amazon S3 bucket. If you
set an Amazon S3 bucket's removal policy to DESTROY, and it contains
data, attempting to destroy the stack will fail because the bucket
cannot be deleted. You can have the AWS CDK delete the objects in the
bucket before attempting to destroy it by setting the bucket's
autoDeleteObjects prop to true.
Best Practice is to
Design stack resources in such a manner that they have minimal updates being applied which can cause failure. So a stack can be created with say mostly static resources such as ECR, S3 which do not change much and is independent generally of the main application deployment stack which is more likely to fail.
Avoid manually deleting the stack resources which breaks a stack's inconsistency
If a stack is deleted, ensure stack's owned resources are also deleted.
Get rid of having fix names!
With
final IBucket myBucket = Bucket.Builder.create(this, "mybucket")
.bucketName(PhysicalName.GENERATE_IF_NEEDED).build();
(Java, but doesn´t matter)
Do you get a "random-Named" Bucket.
Described here: https://docs.aws.amazon.com/cdk/latest/guide/resources.html
Use it like this in your template (here nested stack)
#Nullable NestedStackProps templateProps = NestedStackProps.builder()
.parameters(new HashMap<String, String>(){{
put("S3Bucket", myBucket.getBucketName());
}})
.build();
Or you still have a fix name (get rid of!!) then get them with:
final IBucket myBucket = Bucket.fromBucketName(this, "mybucket", "my-hold-bucket-name");
But you can not doing things like:
if (!myBucket) then create
(pseudo code)
No ressource-check at compile/runtime!
I have created an S3 Bucket, with the cloud formation, Lets Say Bucket Name is S3Bucket,
I don't want this bucket getting deleted if I delete stack, so added Deletion Policy to Retain,
Now the problem here is, If run the stack again, it complains S3Bucket name already exists.
If a bucket already exists, it should not complain.
What to do for this.
Please help
I faced this in the past and what i did in order to resolve this is that i created a common AWS cloudformation template/stack which will create all our common resources which are static(Handle it like a bootstrap template).
Usually i am adding in this template the creation of s3 buckets,VPC, networking, databases creation, etc.
Then you can create other AWS cloudformation templates/stacks for your rest resources which are dynamic and changing usually like lambdas,ec2, api gateway etc.
S3 names are globally unique. (e.g if I have s3 bucket in my AWS account s3-test, you cannot have a bucket with the same name).
The only way to use same name is to delete the bucket, or retype your cloud formation template and use new cloud formation feature to import resource:
https://aws.amazon.com/blogs/aws/new-import-existing-resources-into-a-cloudformation-stack/