Find or create s3 bucket in CDK? - amazon-web-services

Question

I'm finding that cdk tries to recreate S3 buckets every time I deploy. If I don't specify a bucket name, it generates a new junk bucket name every time. If I do specify a name, it refuses to deploy because the bucket already exists. How can I make it "upsert" a bucket?
Here's the code I'm using:
const dataIngestBucket = new Bucket(this, 'data-lake', {
bucketName: `${this.props.environmentName}-my-company-data-lake`
});

Answer 1

As long as I do not see the language you want to use, I give an answer using python. It can be easily traced and converted to any other languages.
Please refer to aws_cdk.aws_s3.Bucket class.
There you will find parameters to specify during class creating which allow you reach your goal, namely auto_delete_objects=True and removal_policy=cdk.RemovalPolicy.DESTROY.

Answer 2

CDK would do an update on your stack resources automatically if CDK code is updated.
For example, when you execute a CDK stack that creates a bucket for the first time, bucket would be created with provided configuration.
When you update your CDK code to say update lifecycle policy of the bucket or add a CORS, as part of the same stack, the update of the stack would automatically update the bucket - it would not recreate the bucket as Cloud Formation knows that there is an update on an existing stack.
In your case, it seems the stack is being re-created after a removal when the stack resources still exists. This causes Cloud Formation to create a new stack and its resources which were not removed when stack was destroyed.
Generally, issues occur when stack update fails and it is in rollback state, for example. In that case, redeploy would try to create the bucket again and fail.
In that case, possible option could be:
Delete the buckets
Delete the stack
Redeploy to re-create
Many times, we do not want to delete the resources, as they contain data; in that case, you can use another library such as boto3 for python in the CDK code, to check if the resource exists - if not create via CDK. This would cause CDK code to be not attempt creating the bucket if it exists ( CDK itself cannot be used to see if say S3 resource exists already - at least have not seen how to achieve this)
Another important point is the removal policy associated with the resource
troubleshooting_resource_not_deleted
My S3 bucket, DynamoDB table, or other resource is not deleted when I
issue cdk destroy
By default, resources that can contain user data have a removalPolicy
(Python: removal_policy) property of RETAIN, and the resource is not
deleted when the stack is destroyed. Instead, the resource is orphaned
from the stack. You must then delete the resource manually after the
stack is destroyed. Until you do, redeploying the stack fails, because
the name of the new resource being created during deployment conflicts
with the name of the orphaned resource.
If you set a resource's removal policy to DESTROY, that resource will
be deleted when the stack is destroyed.
However, even with the removal policy as DESTROY, Cloud formation cannot delete a non-empty bucket. Extract from the same link below -
AWS CloudFormation cannot delete a non-empty Amazon S3 bucket. If you
set an Amazon S3 bucket's removal policy to DESTROY, and it contains
data, attempting to destroy the stack will fail because the bucket
cannot be deleted. You can have the AWS CDK delete the objects in the
bucket before attempting to destroy it by setting the bucket's
autoDeleteObjects prop to true.
Best Practice is to
Design stack resources in such a manner that they have minimal updates being applied which can cause failure. So a stack can be created with say mostly static resources such as ECR, S3 which do not change much and is independent generally of the main application deployment stack which is more likely to fail.
Avoid manually deleting the stack resources which breaks a stack's inconsistency
If a stack is deleted, ensure stack's owned resources are also deleted.

Answer 3

Get rid of having fix names!
With
final IBucket myBucket = Bucket.Builder.create(this, "mybucket")
.bucketName(PhysicalName.GENERATE_IF_NEEDED).build();
(Java, but doesn´t matter)
Do you get a "random-Named" Bucket.
Described here: https://docs.aws.amazon.com/cdk/latest/guide/resources.html
Use it like this in your template (here nested stack)
#Nullable NestedStackProps templateProps = NestedStackProps.builder()
.parameters(new HashMap<String, String>(){{
put("S3Bucket", myBucket.getBucketName());
}})
.build();
Or you still have a fix name (get rid of!!) then get them with:
final IBucket myBucket = Bucket.fromBucketName(this, "mybucket", "my-hold-bucket-name");
But you can not doing things like:
if (!myBucket) then create
(pseudo code)
No ressource-check at compile/runtime!