The new recommendation from AWS is to disable ACL by default such that Object ownership defaults to Bucket owner. How can I achieve this with aws_s3_bucket resource using Terraform?
I tried doing the following without success
resource "aws_s3_bucket_acl" "example_bucket_acl" {
bucket = aws_s3_bucket.s3-bucket.id
acl = "private"
expected_bucket_owner = data.aws_caller_identity.current.account_id
}
data "aws_caller_identity" "current" {}
This code sets the ACL such that only bucket owner can read and write the bucket and the objects within the bucket, but the object ownership configuration is still set to "object writer". Furthermore, ACL is not disabled as a result of setting this.
From the Terraform's documentation on S3 ACL, it does not state any examples nor provide any arguments that support disabling ACL.
I tried to brute force the solution by running terraform plan after manually changing the settings in AWS to see what differences I would get from the plan, but it says my infrastructure matches the configuration.
Does anyone have any ideas how this can be done? I'm currently using Terraform CLI v1.3.5 and AWS provider v4.40.0.
This is set using aws_s3_bucket_ownership_controls, not with aws_s3_bucket_acl. You can set the control to BucketOwnerEnforced.
Related
We create all our AWS resources using Terraform and we are having issues with our S3 bucket.
We are in a development environnement so sometimes we wants to destroy everything and recreate everything except the S3 Bucket.
We tried to implement the following attribute for our S3 Bucket :
resource "aws_s3_bucket" "bucket" {
bucket = "${var.name}-${var.environment}-data"
lifecycle {
prevent_destroy = true
}
}
It seems like the prevent_destroy attribute is not working as we thought it would work. Instead of skipping the deletion of the bucket and terminate the terraform destroy with a success state, it fails instead (as if this attribute tells terraform to fail on purpose).
I've found out similar conclusions already on stackoverflow but what would be the best way to avoid that issue and also the way that would be the best practice please ?
We use also github actions so we have thought of using it to create the bucket but if there's a solution using terraform, it will be easier for us (as there are other resources that are linked to the bucket id).
Thanks in advance !
prevent_destroy is used as a safety measure to ensure that the deletion of the resource does not occur. It is expected that it errors out. From the docs:
This meta-argument, when set to true, will cause Terraform to reject with an error any plan that would destroy the infrastructure object associated with the resource
so you shouldn't be using this meta-argument for your purposes.
Instead, if you want to delete everything except the bucket, you will need to remove the bucket from the terraform state before running the destroy operation, e.g.:
terraform state rm aws_s3_bucket.bucket
terraform destroy
for your next deployment, you will then need to import the bucket back into the terraform state before running the apply:
terraform import aws_s3_bucket.bucket bucket-name
terraform apply
I am trying to deploy a simple flask application on Elastibeanstalk using Terraform.
I am using the Terraform's default resource for ElasticBeanstalk Environment - aws_elastic_beanstalk_environment
I am able to deploy my application successfully, however during deployment ElasticBeanstalk creates an S3 bucket - elasticbeanstalk-region-account-id which is not encrypted by default.
I want to change this behaviour and make sure this bucket is encrypted when it gets created. Which setting do I use to accomplish this? I could not find the relevant setting for this. Any ideas?
by default aws beansltalk create an unencrypted bucket so aws_elastic_beanstalk_environment resource cannot do anything here
from the AWS doc :
Elastic Beanstalk doesn't turn on default encryption for the Amazon S3
bucket that it creates. This means that by default, objects are stored
unencrypted in the bucket (and are accessible only by authorized
users). Some applications require all objects to be encrypted when
they are stored—on a hard drive, in a database, etc. (also known as
encryption at rest). If you have this requirement, you can configure
your account's buckets for default encryption
so you need to enable it yourself, try the folowing
after you create the beanstalk env, get the aws s3 bucket created by beanstalk and enable server side encryption by the Terraform resource aws_s3_bucket_server_side_encryption_configuration
resource "aws_kms_key" "mykey" {
description = "This key is used to encrypt bucket objects"
deletion_window_in_days = 10
}
data "aws_s3_bucket" "mybucket" {
bucket = "elasticbeanstalk-region-account-id" # here change the value with your information
}
resource "aws_s3_bucket_server_side_encryption_configuration" "example" {
bucket = data.aws_s3_bucket.mybucket
rule {
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.mykey.arn
sse_algorithm = "aws:kms"
}
}
}
I created an s3 bucket in terraform. However after creating this bucket, I am getting the error:
error getting S3 Bucket Object Lock configuration: AccessDenied: Access Denied
I am using AWS academy so I do not have many permissions, however, there is a role in AWS academy that allows the user to do stuff with s3. Is there a way to attach this IAM role to the S3 bucket so access it via Terraform?
I would like to upload images to this bucket, however I can no longer deploy code due to Terraform trying to access the Object Lock Configuration which it does not have access to. Is there a way to tell terraform to not try to not try to get this information?
Here is my code
resource "aws_s3_bucket" "b" {
bucket = "my-tf-test-bucket"
acl = "private"
tags = {
Name = "My bucket"
Environment = "Dev"
}
}
Image of Console
So it seems that you have enable object lock to your bucket which prevents you to write or delete any files in your s3 bucket.
One way is to disable it from the console and refresh the state of terraform.
How can I do to use s3 backend that points to a different AWS account?
In other words, I would like to have something like that:
Dev environment state on an S3 bucket in AWS account A
Stage environment state on another S3 bucket on AWS account B
Anyone can help me, please?
The documentation for Terraform's s3 backend includes a section Multi-account AWS Architecture which includes some recommendations, suggestions, and caveats for using Terraform in a multi-account AWS architecture.
That guide is far more detailed than I can reproduce here, but the key points of recommendation are:
Use a separate AWS account for Terraform and any other administrative tools you use to provision and configure your environments, so that the infrastructure that Terraform uses is entirely separate from the infrastructure that Terraform manages.
This reduces the risk of an incorrect Terraform configuration inadvertently breaking your ability to use Terraform itself (e.g. by deleting the state object, or by removing necessary IAM permissions). It also reduces the possibility for an attacker to use vulnerabilities in your main infrastructure to escalate to access to your administrative infrastructure.
Use sts:AssumeRole to indirectly access IAM roles with administrative access in each of your main environment AWS accounts.
This allows you to centralize all of your direct administrative access in a single AWS account where you can more easily audit it, reduces credentials sprawl, and also conveniently configure the AWS provider for that cross-account access (because it has assume_role support built-in).
The guide also discusses using workspaces to represent environments. That advice is perhaps more debatable given the guidance elsewhere in When to use Multiple Workspaces, but the principle of using an administrative account and IAM delegation is still applicable even if you follow this advice of having a separate root module per environment and using shared modules to represent common elements.
As with all things in system architecture, these aren't absolutes and what is best for your case will depend on your details, but hopefully the content in these two documentation sections I've linked to will help you weigh various options and decide what is best for your specific situation.
There are a few solutions to it:
provide aws profile name at the command line while running terraform init and injec terraform backend variables during runtime:
AWS_PROFILE=aws-dev terraform init -backend-config="bucket=825df6bc4eef-state" \
-backend-config="dynamodb_table=825df6bc4eef-state-lock" \
-backend-config="key=terraform-multi-account/terraform.tfstate"
or wrap this command in a Makefile as it is pretty long and forgettable.
Keep separate directories and provide the roles or your credentials or profile name even using shared-credentials
provider "aws" {
region = "us-west-2"
shared_credentials_file = "/Users/tf_user/.aws/creds"
profile = "customprofile"
}
Terraform Workspaces
terragrunt
I don't think it is possible to have a separate S3 backend for each workspace without some hijinks at this time. If you are ok with one S3 backend in one account it's pretty easy to have different accounts associated with each workspace.
# backend.tf
terraform {
backend "s3" {
profile = "default"
bucket = "my-terraform-state"
key = "terraform-multi-account-test/terraform.state"
region = "eu-west-1"
encrypt = true
dynamodb_table = "my-terraform-state-lock"
}
}
and
# provider.tf
variable "workspace_accounts" {
type = map(string)
default = {
"sandbox" = "my-sandbox-keys"
"dev" = "default"
"prod" = "default"
}
}
provider "aws" {
shared_credentials_file = "$HOME/.aws/credentials"
profile = var.workspace_accounts[terraform.workspace]
region = "eu-west-1"
}
See https://github.com/hashicorp/terraform/issues/16627
My understanding is that when configuring an S3 bucket notification with Terraform we can only configure a single notification per S3 bucket:
NOTE: S3 Buckets only support a single notification configuration. Declaring multiple
ws_s3_bucket_notification resources to the same S3 Bucket will cause a perpetual difference in
configuration. See the example "Trigger multiple Lambda functions" for an option.
The application uses a single S3 bucket as a data repository, i.e. when JSON files land there they trigger a lambda which submits a corresponding batch job to ingest from the file into a database.
This works well when we have a single developer deploying the infrastructure, but with multiple developers each time one of us runs terraform apply then it updates the only/single notification for the bucket, overwriting the resource's previous settings.
What is the best practice for utilizing S3 buckets for notifications? Are they best configured/created per Terraform workspace, and/or how are the buckets managed to allow for simultaneous developers standing up/down infrastructure resources using a common S3 bucket via terraform apply, etc.? Must you use one bucket per workspace for this use case, as suggested by the docs?
The current Terraform I have for the S3 notification (the code that allows for overwriting with the latest configuration):
data "aws_s3_bucket" "default" {
bucket = var.bucket
}
resource "aws_lambda_permission" "allow_bucket_execution" {
statement_id = "AllowExecutionFromS3Bucket"
action = "lambda:InvokeFunction"
function_name = var.lambda_function_name
principal = "s3.amazonaws.com"
source_arn = data.aws_s3_bucket.default.arn
}
resource "aws_s3_bucket_notification" "bucket_notification" {
bucket = data.aws_s3_bucket.default.bucket
lambda_function {
lambda_function_arn = var.lambda_function_arn
events = ["s3:ObjectCreated:*"]
filter_prefix = var.namespace
filter_suffix = ".json"
}
}
The namespace variable is passed in as "${local.env}-${terraform.workspace}", with local.env as "dev", "uat", "prod", etc.
How can we modify the Terraform code above to allow for multiple notifications per S3 bucket (essentially one per Terraform workspace), or can it just not be done? If not then how is this best handled? Should I just use a bucket per workspace using a namespace variable like above as the S3 bucket name, and have it updated accordingly to the production bucket at deployment?
There are several options you have depending on your needs:
Create one bucket per env and workspace. Then the mentioned limitation of terraforms aws_s3_bucket_notification should no longer be an issue. I could imagine that the process you use to write to your bucket will then still only write to one bucket you specify. To solve this issue you could think about forwarding any objects uploaded to one "master" bucket to all other buckets (either with a lambda, itself triggered by a aws_s3_bucket_notification or probably by bucket replication).
create one bucket per env and deploy the aws_s3_bucket_notification without workspaces. Then you do no longer have the advantages of workspaces. But this might be a reasonable compromise between number of buckets and usability
keep just this one bucket, keep envs and workspaces, but deploy the aws_s3_bucket_notification resource only once (probably together with the bucket). Then this one aws_s3_bucket_notification resource would need to include the rules for all environment and workspaces.
It really depends on your situation what fits best. If those aws_s3_bucket_notification rarely change at all, and most changes are done in the lambda function, the last option might be the best. If you regularly want to change the aws_s3_bucket_notification and events to listen on, one of the other options might be more suitable.