I have created an S3 Bucket, with the cloud formation, Lets Say Bucket Name is S3Bucket,
I don't want this bucket getting deleted if I delete stack, so added Deletion Policy to Retain,
Now the problem here is, If run the stack again, it complains S3Bucket name already exists.
If a bucket already exists, it should not complain.
What to do for this.
Please help
I faced this in the past and what i did in order to resolve this is that i created a common AWS cloudformation template/stack which will create all our common resources which are static(Handle it like a bootstrap template).
Usually i am adding in this template the creation of s3 buckets,VPC, networking, databases creation, etc.
Then you can create other AWS cloudformation templates/stacks for your rest resources which are dynamic and changing usually like lambdas,ec2, api gateway etc.
S3 names are globally unique. (e.g if I have s3 bucket in my AWS account s3-test, you cannot have a bucket with the same name).
The only way to use same name is to delete the bucket, or retype your cloud formation template and use new cloud formation feature to import resource:
https://aws.amazon.com/blogs/aws/new-import-existing-resources-into-a-cloudformation-stack/
Related
I created an S3 bucket but now I would like to update/add lifecycle policies to it using CDK.
Currently I can import this bucket to a new stack file.
const testBucket = s3.Bucket.fromBucketAttributes(this, 'TestBucket', {
bucketArn: 'arn:aws:s3:::xxxxxx'});
How can I use AwsCustomResource to update/add lifecycle policies? For example, for prefix = long, I want those objects to expire in 7 days, and for prefix = short, I want them to expire in 3 days.
Or is there a general way of updating an existing S3 bucket in a new stack with CDK?
The best option for this is probably to add it to the stack using resource importing. Don't confuse a custom resource with a CDK construct. A custom resource involves deploying a lambda function and calling that function when the CloudFormation custom resource is present in your stack. A CDK construct is used to generate a CloudFormation template. It allows you to combine different resources into a single bit of code, and allows some logical decisions based on the input values.
Steps to import:
Make sure your current CDK is what is deployed. You cannot import resources when there are other changes.
Add the S3 bucket to your CDK code as if you were creating it for the first time. Make sure that all the settings are the same as what is currently deployed. This is important because the import process doesn't validate that what you have configured matches what you are importing.
Run cdk synth to generate the CloudFormation template that includes the S3 bucket.
In the CloudFormation console locate the stack that represents the CDK stack you are working with.
Select Import resources into stack from the Stack actions menu.
Follow the prompts. When asked for the template select Upload a template file and select the template created by the cdk synth command (hint: it will be in cdk-out). You will be prompted to add the bucket name. This should be the bucket that you are wanting to add to this stack.
Once that is done you can modify the bucket as if it were created by your stack.
One thing to note. CDK includes some meta-data in the CloudFormation template. This value must be the same as what is currently deploy or it will be seen as a change and you won't be able to perform the import. You can copy the values from what is currently deployed and manually edit the template created by cdk synth to match that.
Need to access the CfnBucket reference of the testBucket and add lifecycle rules to it.
const testBucket = s3.Bucket.fromBucketAttributes(this, 'TestBucket', {
bucketArn: 'arn:aws:s3:::xxxxxx')
testBucket.node.root.addLifecycleRule({prefix: 'short', expiration:3, enabled:true})
testBucket.node.root.addLifecycleRule({prefix: 'long', expiration:7, enabled:true})
I am trying to create a CloudFormation Template (CFT) for a S3 Bucket that needs to be "PublicRead" and that also has "Requester Pays" turned on.
I have looked at the documentation for S3 Bucket CFTs: AWS::S3::Bucket - AWS CloudFormation
Also I have looked at the documentation for "Requester Pays", but it fails to mention anything about CFTs. It only references enabling it through the console and with the REST API:
Requester Pays Buckets - Amazon Simple Storage Service
Right now we are trying to get all our infrastructure into infrastructure as code, but this is a somewhat large blocker for that. I have heard that other people have had trouble with CFTs not supporting some features from AWS services, but usually those are for unpopular/newer services. I would think that CFT would support all the options that S3 has for buckets.
You are correct. The CloudFormation AWS::S3::Bucket resources does not support Requester Pays.
The enable it, you would need to make an API call such as put_bucket_request_payment():
Sets the request payment configuration for a bucket. By default, the bucket owner pays for downloads from the bucket. This configuration parameter enables the bucket owner (only) to specify that the person requesting the download will be charged for the download.
response = client.put_bucket_request_payment(
Bucket='string',
RequestPaymentConfiguration={
'Payer': 'Requester'|'BucketOwner'
}
)
This could be done by adding an AWS Lambda custom resource to the CloudFormation template, or by using the AWS CLI from an Amazon EC2 instance that is created as part of the stack.
Using my cloudformation template, i was able to create two buckets and one bucket policy in my stack. Turns out my bucket policy had the wrong permissions so i decided to delete the buckets and recreate them with a new template.
It doesn't look like cloudformation has detected my deleted s3 buckets. The buckets still show up in my stack resources but are marked as "Deleted"
My stack is also marked as drifted. When i try to access the s3 buckets via the link in cloudformation, i get "Error Data not found"
My stack has been in this state for about 16 hours. Any idea on how to get cloudformation to sync up with s3?
Your template isn't telling CloudFormation what resources to create, its telling CloudFormation the state that you want.
It sounds like you created a stack with a template with a resource for a bucket.
You then realized a problem and deleted the bucket manually.
You then updated the stack with an updated template with the same resource for the bucket (but with correct permissions)
When CloudFormation processed this updated template, it determined that it had already created the bucket and as a result it didn't recreate it.
You likely could have achieved your desired result without deleting the bucket by just updating the template.
Because you deleted the bucket, your stack is in a bad state. If you have the flexibility to do so, you could delete your stack and recreate it. When you delete it, it may complain about not being able to delete the bucket, you may have to retry once, then get the option to ignore it.
I am still a newbie with AWS services.
I would like to add a Lambda trigger on an existing S3 bucket using a CloudFormation template (CFT). Is this possible?
The following CFT is attempting to create a new S3 bucket and add an event notification on it.
S3BUCKET_NOTIFCATION = Bucket(
"S3Bucket",
BucketName=s3_bucket("confidential", Ref(ENV)),
NotificationConfiguration=NotificationConfiguration(
LambdaConfigurations=[
LambdaConfigurations(
Event="s3:ObjectCreated:*",
Filter=Filter(
S3Key=S3Key(
Rules=[Rules(Name="prefix", Value=Ref(inputKeyPrefix)),
Rules(Name="suffix", Value=".json")]
)
),
Function=Ref(cost_function)
)
]
)
)
Is it possible to add the NotificationConfiguration to an existing bucket?
As noted by others this is still not allowed for various reasons.
Option 1) recommended
The ideal approach here, IMHO, would be to first import existing cloud resources (S3 in this case) to be managed by IaC/CloudFormation. Then, once the resource has been imported, you can add the NotificationConfiguration to your CloudFormation template. This approach allows you to manage existing resources that maybe were initially provisioned manually to be managed via IaC/Cfn moving forward.
Option 2) what you asked for
If you want to add NotificationConfiguration to an existing S3 bucket via CloudFormation the workaround is to use
a Lambda-backed custom resource created... The custom resource
triggers a Lambda function, which triggers the PutBucketNotification
API to add a notification configuration to your S3 bucket.
details here
NOTE: Please take into account the limitations with Option 2). Some of which are documented in the referenced link.
I've been unable to add NotificationConfiguration to an existing bucket as well. When you try, you'll get the error CREATE_FAILED. Reason: S3_BUCKET already exists.
This ServerFault question from 2013 details that modification of a pre-existing bucket is not allowed. It appears to still be correct.
I created S3 bucket using Cloud formation template script.
Now i want to access S3 bucket name and end point from instance metadata.
Any help?
To enable applications running on an Amazon EC2 instance to access Amazon S3 (or any AWS service), you can create an IAM Role for the EC2 instance and assign it to the instance.
Applications on that instance that use the AWS SDK to make API calls to AWS will automatically have access to credentials with permissions described in the assigned role.
Your particular situation is slightly difficult because the CloudFormation template will create a bucket with a unique name, while your IAM Role will want to know the exact name of the Amazon S3 bucket. This can be accomplished by referring to the S3 bucket that was created within the CloudFormation template.
The template would need to create these resources:
AWS::S3::Bucket
AWS::IAM::Role to define the permissions
AWS::IAM::InstanceProfile to link the role to the EC2 instance
AWS::EC2::Instance that refers to the IAM Role
Within the definition of the IAM Role, the ARN for the S3 bucket would need to refer to the bucket created elsewhere in the template. This would require a bit of string manipulation to insert the correct value into the policy.
If you run into difficulty, feel free to create another StackOverflow question showing the template that you have been working on, highlighting the part that is causing difficulty.