I was working on some changes to some CDK code in my AWS account and I made a typo (lack of sleep does some crazy things) in the bucket name (the correct name is something like mybucket-alpha) so it created a new bucket (mybucket-alpha-alpha). I rolled back the change but now I get a "Bucket with name mybucket-alpha already exists" error when I deploy. It seems that the old mybucket-alpha is no longer in the part of my CDK stack, so I get this error because CDK is trying to recreate the bucket. Is there a quick way to reimport this bucket into my stack? I can't delete it since it contains a ton of important data
there is a way to import existing resources into you CDK Stack.
In case of S3 Bucket you can use the Bucket.fromBucketArn() method. You can find it in the documentation as well: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3.Bucket.html#static-fromwbrbucketwbrarnscope-id-bucketarn
Quick example:
const myBucket = Bucket.fromBucketArn(this, 'MyBucket', myBucketArnParam.stringValue);
Here is a thorough explanation of how to import an existing bucket into a cdk stack: https://medium.com/#visya/how-to-import-existing-aws-resources-into-cdk-stack-f1cea491e9
In case the link breaks, the basic steps are...
Add an S3 Bucket to your stack and run cdk synth to generate a template.
In the AWS Console, navigate to the stack to which you wish to add the bucket, choose Stack Actons -> Import resources into stack.
Follow the prompts to upload the template you got via synth
Choose the bucket resource and notice it will pick up the cdk generated id from your template
Complete the import process and then verify the bucket is listed in the stack resources and there is no drift detected.
After importing the bucket into the stack you can managed it via you cdk app/stack.
Related
I created an S3 bucket but now I would like to update/add lifecycle policies to it using CDK.
Currently I can import this bucket to a new stack file.
const testBucket = s3.Bucket.fromBucketAttributes(this, 'TestBucket', {
bucketArn: 'arn:aws:s3:::xxxxxx'});
How can I use AwsCustomResource to update/add lifecycle policies? For example, for prefix = long, I want those objects to expire in 7 days, and for prefix = short, I want them to expire in 3 days.
Or is there a general way of updating an existing S3 bucket in a new stack with CDK?
The best option for this is probably to add it to the stack using resource importing. Don't confuse a custom resource with a CDK construct. A custom resource involves deploying a lambda function and calling that function when the CloudFormation custom resource is present in your stack. A CDK construct is used to generate a CloudFormation template. It allows you to combine different resources into a single bit of code, and allows some logical decisions based on the input values.
Steps to import:
Make sure your current CDK is what is deployed. You cannot import resources when there are other changes.
Add the S3 bucket to your CDK code as if you were creating it for the first time. Make sure that all the settings are the same as what is currently deployed. This is important because the import process doesn't validate that what you have configured matches what you are importing.
Run cdk synth to generate the CloudFormation template that includes the S3 bucket.
In the CloudFormation console locate the stack that represents the CDK stack you are working with.
Select Import resources into stack from the Stack actions menu.
Follow the prompts. When asked for the template select Upload a template file and select the template created by the cdk synth command (hint: it will be in cdk-out). You will be prompted to add the bucket name. This should be the bucket that you are wanting to add to this stack.
Once that is done you can modify the bucket as if it were created by your stack.
One thing to note. CDK includes some meta-data in the CloudFormation template. This value must be the same as what is currently deploy or it will be seen as a change and you won't be able to perform the import. You can copy the values from what is currently deployed and manually edit the template created by cdk synth to match that.
Need to access the CfnBucket reference of the testBucket and add lifecycle rules to it.
const testBucket = s3.Bucket.fromBucketAttributes(this, 'TestBucket', {
bucketArn: 'arn:aws:s3:::xxxxxx')
testBucket.node.root.addLifecycleRule({prefix: 'short', expiration:3, enabled:true})
testBucket.node.root.addLifecycleRule({prefix: 'long', expiration:7, enabled:true})
How can I get the s3 bucket name from the bootstrap stack?
I cant seems to find any other method than creating a dummy asset and get the s3BucketName attribute.
--
Update:
I have two stacks:
Stack 1: Creating a bunch of S3 assets.
Stack 2: Creating an IAM policy to give access to the assests created in Stack 1 (no assets)
I need to lookup the bootstrapped assets bucket in Stack 2 to be able to add the resources attribute in my IAM Policy
I find multiple solutions for this, but all of them seems hacky
The CDK framework can reference the bucket when instantiating new assets, without me providing the bucket name, this implies that the framework know the bucket name, hence it must be possible for me to get hold of it and avoid either hardcoding, creating dummy assets or using Fn::Import when adding the resources attribute in my IAM Policy in Stack 2
To answer my own question, you can get the asset bucket name through the DefaultStackSynthesizer:
example:
const assetBucketName = cdk.DefaultStackSynthesizer.DEFAULT_FILE_ASSETS_BUCKET_NAME
Toolkit reference
cdk bootstrap --toolkit-bucket-name <bucket name>
I'm finding that cdk tries to recreate S3 buckets every time I deploy. If I don't specify a bucket name, it generates a new junk bucket name every time. If I do specify a name, it refuses to deploy because the bucket already exists. How can I make it "upsert" a bucket?
Here's the code I'm using:
const dataIngestBucket = new Bucket(this, 'data-lake', {
bucketName: `${this.props.environmentName}-my-company-data-lake`
});
As long as I do not see the language you want to use, I give an answer using python. It can be easily traced and converted to any other languages.
Please refer to aws_cdk.aws_s3.Bucket class.
There you will find parameters to specify during class creating which allow you reach your goal, namely auto_delete_objects=True and removal_policy=cdk.RemovalPolicy.DESTROY.
CDK would do an update on your stack resources automatically if CDK code is updated.
For example, when you execute a CDK stack that creates a bucket for the first time, bucket would be created with provided configuration.
When you update your CDK code to say update lifecycle policy of the bucket or add a CORS, as part of the same stack, the update of the stack would automatically update the bucket - it would not recreate the bucket as Cloud Formation knows that there is an update on an existing stack.
In your case, it seems the stack is being re-created after a removal when the stack resources still exists. This causes Cloud Formation to create a new stack and its resources which were not removed when stack was destroyed.
Generally, issues occur when stack update fails and it is in rollback state, for example. In that case, redeploy would try to create the bucket again and fail.
In that case, possible option could be:
Delete the buckets
Delete the stack
Redeploy to re-create
Many times, we do not want to delete the resources, as they contain data; in that case, you can use another library such as boto3 for python in the CDK code, to check if the resource exists - if not create via CDK. This would cause CDK code to be not attempt creating the bucket if it exists ( CDK itself cannot be used to see if say S3 resource exists already - at least have not seen how to achieve this)
Another important point is the removal policy associated with the resource
troubleshooting_resource_not_deleted
My S3 bucket, DynamoDB table, or other resource is not deleted when I
issue cdk destroy
By default, resources that can contain user data have a removalPolicy
(Python: removal_policy) property of RETAIN, and the resource is not
deleted when the stack is destroyed. Instead, the resource is orphaned
from the stack. You must then delete the resource manually after the
stack is destroyed. Until you do, redeploying the stack fails, because
the name of the new resource being created during deployment conflicts
with the name of the orphaned resource.
If you set a resource's removal policy to DESTROY, that resource will
be deleted when the stack is destroyed.
However, even with the removal policy as DESTROY, Cloud formation cannot delete a non-empty bucket. Extract from the same link below -
AWS CloudFormation cannot delete a non-empty Amazon S3 bucket. If you
set an Amazon S3 bucket's removal policy to DESTROY, and it contains
data, attempting to destroy the stack will fail because the bucket
cannot be deleted. You can have the AWS CDK delete the objects in the
bucket before attempting to destroy it by setting the bucket's
autoDeleteObjects prop to true.
Best Practice is to
Design stack resources in such a manner that they have minimal updates being applied which can cause failure. So a stack can be created with say mostly static resources such as ECR, S3 which do not change much and is independent generally of the main application deployment stack which is more likely to fail.
Avoid manually deleting the stack resources which breaks a stack's inconsistency
If a stack is deleted, ensure stack's owned resources are also deleted.
Get rid of having fix names!
With
final IBucket myBucket = Bucket.Builder.create(this, "mybucket")
.bucketName(PhysicalName.GENERATE_IF_NEEDED).build();
(Java, but doesn´t matter)
Do you get a "random-Named" Bucket.
Described here: https://docs.aws.amazon.com/cdk/latest/guide/resources.html
Use it like this in your template (here nested stack)
#Nullable NestedStackProps templateProps = NestedStackProps.builder()
.parameters(new HashMap<String, String>(){{
put("S3Bucket", myBucket.getBucketName());
}})
.build();
Or you still have a fix name (get rid of!!) then get them with:
final IBucket myBucket = Bucket.fromBucketName(this, "mybucket", "my-hold-bucket-name");
But you can not doing things like:
if (!myBucket) then create
(pseudo code)
No ressource-check at compile/runtime!
I have created an S3 Bucket, with the cloud formation, Lets Say Bucket Name is S3Bucket,
I don't want this bucket getting deleted if I delete stack, so added Deletion Policy to Retain,
Now the problem here is, If run the stack again, it complains S3Bucket name already exists.
If a bucket already exists, it should not complain.
What to do for this.
Please help
I faced this in the past and what i did in order to resolve this is that i created a common AWS cloudformation template/stack which will create all our common resources which are static(Handle it like a bootstrap template).
Usually i am adding in this template the creation of s3 buckets,VPC, networking, databases creation, etc.
Then you can create other AWS cloudformation templates/stacks for your rest resources which are dynamic and changing usually like lambdas,ec2, api gateway etc.
S3 names are globally unique. (e.g if I have s3 bucket in my AWS account s3-test, you cannot have a bucket with the same name).
The only way to use same name is to delete the bucket, or retype your cloud formation template and use new cloud formation feature to import resource:
https://aws.amazon.com/blogs/aws/new-import-existing-resources-into-a-cloudformation-stack/
I am still a newbie with AWS services.
I would like to add a Lambda trigger on an existing S3 bucket using a CloudFormation template (CFT). Is this possible?
The following CFT is attempting to create a new S3 bucket and add an event notification on it.
S3BUCKET_NOTIFCATION = Bucket(
"S3Bucket",
BucketName=s3_bucket("confidential", Ref(ENV)),
NotificationConfiguration=NotificationConfiguration(
LambdaConfigurations=[
LambdaConfigurations(
Event="s3:ObjectCreated:*",
Filter=Filter(
S3Key=S3Key(
Rules=[Rules(Name="prefix", Value=Ref(inputKeyPrefix)),
Rules(Name="suffix", Value=".json")]
)
),
Function=Ref(cost_function)
)
]
)
)
Is it possible to add the NotificationConfiguration to an existing bucket?
As noted by others this is still not allowed for various reasons.
Option 1) recommended
The ideal approach here, IMHO, would be to first import existing cloud resources (S3 in this case) to be managed by IaC/CloudFormation. Then, once the resource has been imported, you can add the NotificationConfiguration to your CloudFormation template. This approach allows you to manage existing resources that maybe were initially provisioned manually to be managed via IaC/Cfn moving forward.
Option 2) what you asked for
If you want to add NotificationConfiguration to an existing S3 bucket via CloudFormation the workaround is to use
a Lambda-backed custom resource created... The custom resource
triggers a Lambda function, which triggers the PutBucketNotification
API to add a notification configuration to your S3 bucket.
details here
NOTE: Please take into account the limitations with Option 2). Some of which are documented in the referenced link.
I've been unable to add NotificationConfiguration to an existing bucket as well. When you try, you'll get the error CREATE_FAILED. Reason: S3_BUCKET already exists.
This ServerFault question from 2013 details that modification of a pre-existing bucket is not allowed. It appears to still be correct.