Creating a bucket is pretty simple.
resource "aws_s3_bucket" "henrys_bucket" {
bucket = "${var.s3_bucket_name}"
acl = "private"
force_destroy = "true"
}
Initially I thought I could create a list for the s3_bucket_name variable but I get an error:
Error: bucket must be a single value, not a list
-
variable "s3_bucket_name" {
type = "list"
default = ["prod_bucket", "stage-bucket", "qa_bucket"]
}
How can I create multiple buckets without duplicating code?
You can use a combination of count & element like so:
variable "s3_bucket_name" {
type = "list"
default = ["prod_bucket", "stage-bucket", "qa_bucket"]
}
resource "aws_s3_bucket" "henrys_bucket" {
count = "${length(var.s3_bucket_name)}"
bucket = "${element(var.s3_bucket_name, count.index)}"
acl = "private"
force_destroy = "true"
}
Edit: as suggested by #ydaetskcoR you can use the list[index] pattern rather than element.
variable "s3_bucket_name" {
type = "list"
default = ["prod_bucket", "stage-bucket", "qa_bucket"]
}
resource "aws_s3_bucket" "henrys_bucket" {
count = "${length(var.s3_bucket_name)}"
bucket = "${var.s3_bucket_name[count.index]}"
acl = "private"
force_destroy = "true"
}
Related
I have the following module which works fine.
Module
resource "aws_s3_bucket" "buckets" {
bucket = var.s3_buckets
}
resource "aws_s3_bucket_acl" "buckets" {
bucket = var.s3_buckets
acl = "private"
}
Root module
module "s3_buckets" {
source = "./modules/s3"
for_each = toset([
"bucket-test1-${var.my_env}",
"bucket-test2-${var.my_env}",
])
s3_buckets = each.value
}
I would like to add get the following policy to all the buckets in the list. Obviously the count option below does not work.
data "aws_iam_policy_document" "buckets" {
count = length(var.s3_buckets)
statement {
sid = "AllowSSlRequestsOnly"
actions = ["s3:*"]
effect = "Deny"
condition {
test = "Bool"
variable = "aws:SecureTransport"
values = ["false"]
}
principals {
type = "*"
identifiers = ["*"]
}
resources = ["arn:aws:s3:::${var.s3_buckets}"]
}
}
resource "aws_s3_bucket_policy" "buckets" {
bucket = var.s3_buckets
policy = data.aws_iam_policy_document.buckets[count.index].json
}
I'm thinking that I need another for_each and use a resource for the IAM policy, I have seen an example such as below, but in the current form I'm providing a string instead a set of strings. Any ideas?
resource "aws_s3_bucket_policy" "buckets" {
for_each = var.s3_buckets
bucket = each.key
policy = jsonencode({
Version = "2012-10-17"
Id = "AllowSSlRequestsOnly",
Statement = [
{
Sid = "AllowSSlRequestsOnly"
Effect = "Deny"
Principal = "*"
Action = "s3:*"
Resource = each.value.arn
Condition = {
Bool = {
"aws:SecureTransport": "false"
}
}
}
]
})
}
If you are adding the policy inside the module (which you probably are, otherwise it doesn't make much sense to attach the policies outside, since you have full control) - then why do you need to mingle with count() at all?
Create the policy and attach to the bucket like:
data "aws_iam_policy_document" "buckets" {
statement {
sid = "AllowSSlRequestsOnly"
actions = ["s3:*"]
effect = "Deny"
condition {
test = "Bool"
variable = "aws:SecureTransport"
values = ["false"]
}
principals {
type = "*"
identifiers = ["*"]
}
resources = ["arn:aws:s3:::${var.s3_buckets}"]
}
}
resource "aws_s3_bucket_policy" "buckets" {
bucket = var.s3_buckets
policy = data.aws_iam_policy_document.buckets.json
}
Couple more points:
It's a singular bucket that you are passing to the module, yet the variable is named s3_buckets, it is confusing.
Using var.s3_buckets for all dependent resources is not the best practice. Create the bucket with var.s3.buckets, after which use the outputs of the resource. This and examples of policies reside here: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket
good luck ☺️
We are currently in the process of editing our core s3 module to adapt to the new V4.0 changes that terraform released -
Existing main.tf
bucket = local.bucket_name
tags = local.tags
force_destroy = var.force_destroy
dynamic "logging" {
for_each = local.logging
content {
target_bucket = logging.value["target_bucket"]
target_prefix = logging.value["target_prefix"]
}
}
....
}
I am trying to convert this to use the resource aws_s3_bucket_logging as below
resource "aws_s3_bucket" "bucket" {
bucket = local.bucket_name
tags = local.tags
force_destroy = var.force_destroy
hosted_zone_id = var.hosted_zone_id
}
resource "aws_s3_bucket_logging" "logging" {
bucket = aws_s3_bucket.bucket.id
dynamic "logging" {
for_each = local.logging
content {
target_bucket = logging.value["target_bucket"]
target_prefix = logging.value["target_prefix"]
}
}
locals.tf
locals {
logging = var.log_bucket == null ? [] : [
{
target_bucket = var.log_bucket
target_prefix = var.log_prefix
}
]
....
variables.tf
type = string
default = null
description = "The name of the bucket that will receive the log objects."
}
variable "log_prefix" {
type = string
default = null
description = "To specify a key prefix for log objects."
}
And I receive the error
Error: Unsupported block type
Blocks of type "logging" are not expected here.
Any help is greatly appreciated. TA
If you check TF docs aws_s3_bucket_logging, you will find that aws_s3_bucket_logging does not have any block nor attribute called logging. Please have a look at the docs linked, and follow the examples and the documentation.
Although, Is it the right usage of the resource if i remove the locals and simplify the resource to be as below? All am i trying to is pass null as default value
resource "aws_s3_bucket_logging" "logging" {
bucket = aws_s3_bucket.bucket.id
target_bucket = var.log_bucket
target_prefix = var.log_prefix
}
I'd like to iterate over a map variable and concatenate individual strings to a single string for an S3 subfolder.
My expected output is:
s3://my-bucket/name1/k1/abc/
s3://my-bucket/name1/k2/def/
s3://my-bucket/name2/k3/xyz/
But with the current setup I get a Cannot include the given value in a string template: string required. Is there another way to loop over my source_names variable or should I set it up differently?
main.tf s3_buckets modules
resource "aws_s3_bucket" "bucket" {
bucket = var.bucket_name
acl = "private"
tags = {
Name = var.tag
}
}
resource "aws_s3_object" "new_folders" {
for_each = var.source_names
bucket = aws_s3_bucket.bucket.id
acl = "private"
key = "${each.key}/${each.value.key_name}/${each.value.key_value}"
}
resource "aws_s3_bucket_public_access_block" "example" {
bucket = aws_s3_bucket.bucket.id
block_public_acls = true
block_public_policy = true
}
variables.tf s3_buckets_module
variable "bucket_name" {
description = "Name of the bucket"
type = string
}
variable "tag" {
description = "Resource tag"
type = string
}
variable "source_names" {
description = "key"
type = map(object({
key_name = list(string)
key_value = list(string)
}))
}
main.tf pipeline module
module "s3_bucket" {
source = "../s3_buckets"
bucket_name = "tf-list-keys-bucket"
tag = "tf"
source_names = {"name1" = {"key_name" = ["k1", "k2"], "key_value" = ["abc/", "def/"]},
"name2" = {"key_name" = ["k3"], "key_value" = ["xyz/"]}
}
}
main.tf
module "pipeline" {
source = "../modules/pipeline"
}
Iterating over your source_names requires you to have 2 embedder for loops:
resource "aws_s3_object" "new_folders" {
for_each = toset(flatten([
for key, value in var.source_names : [
for key_name, key_values in value : [
for key_value in key_values :
"${key}/${key_name}/${key_value}"
]]
]))
bucket = aws_s3_bucket.bucket.id
acl = "private"
key = each.value
}
You have to flatten out your data structure, otherwise Terraform wont know how many resources to provision. You cannot just put an array in a string and expecting Terraform to iterate over it (example: key = "${each.key}/${each.value.key_name}/${each.value.key_value}" - each.value.key_value is an array).
main.tf
module "iam_assumable_role" {
for_each = var.service_accounts
source = "../../../../../../modules/iam-assumable-role-with-oidc/"
create_role = true
role_name = each.value.name
provider_url = replace(module.eks.cluster_oidc_issuer_url, "https://", "")
// role_policy_arns = [for i in each.value.policies : "aws_iam_policy.${i}.arn"]
oidc_fully_qualified_subjects = each.value.wildcard == "" ? ["system:serviceaccount:${each.value.namespace}:${each.value.name}"] : []
oidc_subjects_with_wildcards = each.value.wildcard != "" ? ["system:serviceaccount:${each.value.namespace}:${each.value.wildcard}"] : []
tags = var.tags
}
resource "aws_iam_policy" "dev-policy1" {
name_prefix = "dev-policy"
description = "some description"
policy = data.aws_iam_policy_document.dev-policy1.json
}
variable "service_accounts" {
type = map(object({
name = string
namespace = string
wildcard = string
policies = list(any)
}))
}
tfvars
service_accounts = {
"dev-sa" = {
"name" = "dev-sa",
"namespace" = "dev",
"wildcard" = "*",
"policies" = ["dev-policy1", "dev-policy2"]
},
"qa-sa" = {
"name" = "qa-sa",
"namespace" = "qa",
"wildcard" = "*",
"policies" = ["qa-policy1", "qa-policy2"]
}
}
My code is iterating over service_accounts variable and creates appropriate resources. The problem is that in the commented line I cannot get the list of aws_iam_policy.arn s for the provided policy names (policy names are provided through service_account variable). My current code returns the aws_iam_policy.PolicyName.arn as string and not the actual value. Note that dev-policy1 resource s just one of the all policy resources. All policy documents exist as well. module itself is working correctly when I provide policy list directly and not through variable.
Is it possible to achieve the desired in terraform at all?
You have to use for_each, to create your policies, as you can't dynamically references individual resources the way you are trying to do:
# get all policy names. Your names are unique, so its fine to use list
locals {
policy_names = flatten(values(var.service_accounts)[*]["policies"])
}
# create policy for each name in `policy_names`
resource "aws_iam_policy" "policy" {
for_each = local.policy_names
name_prefix = "dev-policy"
description = "some description"
# similar must be done below
# policy = data.aws_iam_policy_document.dev-policy1.json
}
Then you refer to them as:
role_policy_arns = [for i in each.value.policies: aws_iam_policy[${i}].arn]
I am looking to create one S3 terraform module which can take list of bucket names and folder names to be created inside all those buckets.
For e.g. in my S3 module main.tf. I have
resource "aws_s3_bucket_object" "folders" {
count = var.create_folders ? length(var.s3_folder_names) : 0
bucket = element(aws_s3_bucket.s3bucket.*.id, count.index)
acl = "private"
key = format("%s/",var.s3_folder_names[count.index])
source = "/dev/null"
}
I am calling this module as given below:
variable "s3_bucket_name" {
type = list
description = "List of S3 bucket names"
default = ["bucket1","bucket-2"]
}
variable "s3_folder_names" {
type = list
description = "The list of S3 folders to be created inside S3 bucket"
default=["folder1/dev","folder2/qa"]
}
module "s3" {
source = "../../../gce-nextgen-cloud-terraform-modules/modules/s3"
create_folders = true
s3_folder_names = var.s3_folder_names
environment = var.environment
s3_bucket_name = var.s3_bucket_name
force_destroy = true
bucket_replication_enabled = true
tags = local.common_tags
providers = {
aws.main_region = aws.main_region
aws.secondary_region = aws.secondary_region
}
}
I am facing problem because count variable can only be set once in resource block. Here is the scenario that is cauing problems:
If
var.s3_folder_names < aws_s3_bucket.s3bucket.*.id.
Then I will not be able to access all the elements of S3 bucket list as shown below
resource "aws_s3_bucket_object" "folders" {
count = var.create_folders ? length(var.s3_folder_names) : 0
**bucket = element(aws_s3_bucket.s3bucket.*.id, count.index)**
acl = "private"
key = format("%s/",var.s3_folder_names[count.index])
source = "/dev/null"
}
Hence because of this I will not be able to create these folders inside all of the buckets. The only goal is to create same set of folder structure within all of those buckets.
Any help would be truly appreciated. Thanks in advance!
You can create a combined data structure, e.g.:
locals {
buckets_and_folders = merge([
for bucket in var.s3_bucket_name:
{
for folder in var.s3_folder_names:
"${bucket}-${folder}" => {
bucket = bucket
folder = folder
}
}
]...)
}
Then you would iterate over this structure using for_each:
resource "aws_s3_bucket_object" "folders" {
for_each = var.create_folders ? local.buckets_and_folders : {}
bucket = each.value.bucket
acl = "private"
key = format("%s/", each.value.folder)
source = "/dev/null"
}