Create multiple folders within multiple S3 buckets with Terraform - amazon-web-services

I am looking to create one S3 terraform module which can take list of bucket names and folder names to be created inside all those buckets.
For e.g. in my S3 module main.tf. I have
resource "aws_s3_bucket_object" "folders" {
count = var.create_folders ? length(var.s3_folder_names) : 0
bucket = element(aws_s3_bucket.s3bucket.*.id, count.index)
acl = "private"
key = format("%s/",var.s3_folder_names[count.index])
source = "/dev/null"
}
I am calling this module as given below:
variable "s3_bucket_name" {
type = list
description = "List of S3 bucket names"
default = ["bucket1","bucket-2"]
}
variable "s3_folder_names" {
type = list
description = "The list of S3 folders to be created inside S3 bucket"
default=["folder1/dev","folder2/qa"]
}
module "s3" {
source = "../../../gce-nextgen-cloud-terraform-modules/modules/s3"
create_folders = true
s3_folder_names = var.s3_folder_names
environment = var.environment
s3_bucket_name = var.s3_bucket_name
force_destroy = true
bucket_replication_enabled = true
tags = local.common_tags
providers = {
aws.main_region = aws.main_region
aws.secondary_region = aws.secondary_region
}
}
I am facing problem because count variable can only be set once in resource block. Here is the scenario that is cauing problems:
If
var.s3_folder_names < aws_s3_bucket.s3bucket.*.id.
Then I will not be able to access all the elements of S3 bucket list as shown below
resource "aws_s3_bucket_object" "folders" {
count = var.create_folders ? length(var.s3_folder_names) : 0
**bucket = element(aws_s3_bucket.s3bucket.*.id, count.index)**
acl = "private"
key = format("%s/",var.s3_folder_names[count.index])
source = "/dev/null"
}
Hence because of this I will not be able to create these folders inside all of the buckets. The only goal is to create same set of folder structure within all of those buckets.
Any help would be truly appreciated. Thanks in advance!

You can create a combined data structure, e.g.:
locals {
buckets_and_folders = merge([
for bucket in var.s3_bucket_name:
{
for folder in var.s3_folder_names:
"${bucket}-${folder}" => {
bucket = bucket
folder = folder
}
}
]...)
}
Then you would iterate over this structure using for_each:
resource "aws_s3_bucket_object" "folders" {
for_each = var.create_folders ? local.buckets_and_folders : {}
bucket = each.value.bucket
acl = "private"
key = format("%s/", each.value.folder)
source = "/dev/null"
}

Related

Terraform "Unsupported Block type" error when using aws_s3_bucket_logging resource

We are currently in the process of editing our core s3 module to adapt to the new V4.0 changes that terraform released -
Existing main.tf
bucket = local.bucket_name
tags = local.tags
force_destroy = var.force_destroy
dynamic "logging" {
for_each = local.logging
content {
target_bucket = logging.value["target_bucket"]
target_prefix = logging.value["target_prefix"]
}
}
....
}
I am trying to convert this to use the resource aws_s3_bucket_logging as below
resource "aws_s3_bucket" "bucket" {
bucket = local.bucket_name
tags = local.tags
force_destroy = var.force_destroy
hosted_zone_id = var.hosted_zone_id
}
resource "aws_s3_bucket_logging" "logging" {
bucket = aws_s3_bucket.bucket.id
dynamic "logging" {
for_each = local.logging
content {
target_bucket = logging.value["target_bucket"]
target_prefix = logging.value["target_prefix"]
}
}
locals.tf
locals {
logging = var.log_bucket == null ? [] : [
{
target_bucket = var.log_bucket
target_prefix = var.log_prefix
}
]
....
variables.tf
type = string
default = null
description = "The name of the bucket that will receive the log objects."
}
variable "log_prefix" {
type = string
default = null
description = "To specify a key prefix for log objects."
}
And I receive the error
Error: Unsupported block type
Blocks of type "logging" are not expected here.
Any help is greatly appreciated. TA
If you check TF docs aws_s3_bucket_logging, you will find that aws_s3_bucket_logging does not have any block nor attribute called logging. Please have a look at the docs linked, and follow the examples and the documentation.
Although, Is it the right usage of the resource if i remove the locals and simplify the resource to be as below? All am i trying to is pass null as default value
resource "aws_s3_bucket_logging" "logging" {
bucket = aws_s3_bucket.bucket.id
target_bucket = var.log_bucket
target_prefix = var.log_prefix
}

Terraform - Copy multiple Files to bucket at the same time bucket creation

Hello,
I have a little head hache.
I want to create buckets and cp bulk files at the same time. I have multiple folder (datasetname) in schema folder with json file: schema/dataset1 schema/dataset2 schema/dataset3
The trick is Terraform generate bucketname + random numbers to avoid already name used. I have one question:
How to copy bulk files in a bucket (at the same time bucket creation)
resource "google_storage_bucket" "map" {
for_each = {for i, v in var.gcs_buckets: i => v}
name = "${each.value.id}_${random_id.suffix[0].hex}"
location = var.default_region
storage_class = "REGIONAL"
uniform_bucket_level_access = true
#If you destroy your bucket, this option will delete all objects inside this bucket
#if not Terrafom will fail that run
force_destroy = true
labels = {
env = var.env_label
}
resource "google_storage_bucket_object" "map" {
for_each = {for i, v in var.json_buckets: i => v}
name = ""
source = "schema/${each.value.dataset_name}/*"
bucket = contains([each.value.bucket_name], each.value.dataset_name)
#bucket = "${google_storage_bucket.map[contains([each.value.bucket_name], each.value.dataset_name)]}"
}
variable "json_buckets" {
type = list(object({
bucket_name = string
dataset_name = string
}))
default = [
{
bucket_name = "schema_table1",
dataset_name = "dataset1",
},
{
bucket_name = "schema_table2",
dataset_name = "dataset2",
},
{
bucket_name = "schema_table2",
dataset_name = "dataset3",
},
]
}
variable "gcs_buckets" {
type = list(object({
id = string
description = string
}))
default = [
{
id = "schema_table1",
description = "schema_table1",
},
]
}
...
Why do you have bucket = contains([each.value.bucket_name], each.value.dataset_name)? The contains function returns a bool, and bucket takes a string input (the name of the bucket).
There is no resource that will allow you to copy multiple objects at once to the bucket. If you need to do this in Terraform, you can use the fileset function to get a list of files in your directory, then use that list in your for_each for the google_storage_bucket_object. It might look something like this (untested):
locals {
// Create a master list that has all files for all buckets
all_files = merge([
// Loop through each bucket/dataset combination
for bucket_idx, bucket_data in var.json_buckets:
{
// For each bucket/dataset combination, get a list of all files in that dataset
for file in fileset("schema/${bucket_data.dataset_name}/", "**"):
// And stick it in a map of all bucket/file combinations
"bucket-${bucket_idx}-${file}" => merge(bucket_data, {
file_name = file
})
}
]...)
}
resource "google_storage_bucket_object" "map" {
for_each = local.all_files
name = each.value.file_name
source = "schema/${each.value.dataset_name}/${each.value.file_name}"
bucket = each.value.bucket_name
}
WARNING: Do not do this if you have a lot of files to upload. This will create a resource in the Terraform state file for each uploaded file, meaning every time you run terraform plan or terraform apply, it will do an API call to check the status of each uploaded file. It will get very slow very quickly if you have hundreds of files to upload.
If you have a ton of files to upload, consider using an external CLI-based tool to sync the local files with the remote bucket after the bucket is created. You can use a module such as this one to run external CLI commands.

How can I attach a generic policy to more than 1 bucket, while having the bucket arn dynamically?

I am learning terraform and currently attempting to attach a policy to a created bucket. More specifically, the policy I want to attach has the same permissions/structure but the only difference is the resources section. I will illustrate with an example:
Let's say I create an s3 bucket like:
module "happy_bucket" {
source = "outer space"
name = "happy-bucket"
}
And another bucket like:
module "sad_bucket" {
source = "outer space"
name = "sad-bucket"
}
And now I have a policy that looks like:
data "aws_iam_policy_document" "some_policy" {
statement {
effect = "Allow"
resources = [module.some_bucket.bucket_arn]
actions = ["s3:GetObject", "s3:GetObjectVersion"]
}
}
And now I would like to attach "some_policy" to both "sad-bucket" and "happy-bucket". But I want to do that without having to repeat myself by creating the policy two times (because I need the .bucket_arn to be based on the bucket). In other words, I want to create one generic policy, and attach it to the 2 buckets I created (while picking up the arn dynamically).
Bucket policies require principals, so you need to add that to your some_policy. Having said that, if you want to keep using aws_iam_policy_document you use for_each to iterate over your buckets.
For example, if your module is:
variable "name" {}
resource "aws_s3_bucket" "b" {
bucket = var.name
}
output "bucket_arn" {
value = aws_s3_bucket.b.arn
}
output "bucket_name" {
value = aws_s3_bucket.b.id
}
then in parent module, you can:
module "happy_bucket" {
source = "./modules/buckets"
name = "happy-bucket-231123124ff"
}
module "sad_bucket" {
source = "./modules/buckets"
name = "sad-bucket-231123124ff"
}
locals {
my_buckets = {for bucket in [module.happy_bucket, module.sad_bucket]:
bucket.bucket_name => bucket}
}
data "aws_iam_policy_document" "some_policy" {
for_each = local.my_buckets
statement {
effect = "Allow"
resources = ["${each.value.bucket_arn}/*"]
actions = ["s3:GetObject", "s3:GetObjectVersion"]
principals {
type = "AWS"
identifiers = ["*"]
}
}
}
resource "aws_s3_bucket_policy" "bucket_policie" {
for_each = local.my_buckets
bucket = each.key
policy = data.aws_iam_policy_document.some_policy[each.key].json
}

How to avoid cycle error when setting an S3 bucket policy with a template that depends on the bucket name?

I have a terraform file which fails when I run terraform plan and I get the error:
Error: Cycle: module.hosting.data.template_file.bucket_policy, module.hosting.aws_s3_bucket.website
It makes sense since the bucket refers to the policy and vice versa:
data "template_file" "bucket_policy" {
template = file("${path.module}/policy.json")
vars = {
bucket = aws_s3_bucket.website.arn
}
}
resource "aws_s3_bucket" "website" {
bucket = "xxx-website"
website {
index_document = "index.html"
}
policy = data.template_file.bucket_policy.rendered
}
How can I avoid this bidirectional reference?
You can use the aws_s3_bucket_policy resource. This allows you to create the resources without a circular dependency.
This way, Terraform can:
Create the bucket
Create the template file, using the bucket ARN
Create the policy, referring back to the template file and attaching it to the bucket.
The code would look something like this:
data "template_file" "bucket_policy" {
template = file("${path.module}/policy.json")
vars = {
bucket = aws_s3_bucket.website.arn
}
}
resource "aws_s3_bucket" "website" {
bucket = "xxx-website"
website {
index_document = "index.html"
}
}
resource "aws_s3_bucket_policy" "b" {
bucket = "${aws_s3_bucket.website.id}"
policy = data.template_file.bucket_policy.rendered
}
You could build the ARN of the bucket yourself:
locals {
bucket_name = "example"
bucket_arn = "arn:aws:s3:::${local.bucket_name}"
}
data "template_file" "bucket_policy" {
template = file("${path.module}/policy.json")
vars = {
bucket = local.bucket_arn
}
}
resource "aws_s3_bucket" "website" {
bucket = local.bucket_name
website {
index_document = "index.html"
}
policy = data.template_file.bucket_policy.rendered
}

Terraform - creating multiple buckets

Creating a bucket is pretty simple.
resource "aws_s3_bucket" "henrys_bucket" {
bucket = "${var.s3_bucket_name}"
acl = "private"
force_destroy = "true"
}
Initially I thought I could create a list for the s3_bucket_name variable but I get an error:
Error: bucket must be a single value, not a list
-
variable "s3_bucket_name" {
type = "list"
default = ["prod_bucket", "stage-bucket", "qa_bucket"]
}
How can I create multiple buckets without duplicating code?
You can use a combination of count & element like so:
variable "s3_bucket_name" {
type = "list"
default = ["prod_bucket", "stage-bucket", "qa_bucket"]
}
resource "aws_s3_bucket" "henrys_bucket" {
count = "${length(var.s3_bucket_name)}"
bucket = "${element(var.s3_bucket_name, count.index)}"
acl = "private"
force_destroy = "true"
}
Edit: as suggested by #ydaetskcoR you can use the list[index] pattern rather than element.
variable "s3_bucket_name" {
type = "list"
default = ["prod_bucket", "stage-bucket", "qa_bucket"]
}
resource "aws_s3_bucket" "henrys_bucket" {
count = "${length(var.s3_bucket_name)}"
bucket = "${var.s3_bucket_name[count.index]}"
acl = "private"
force_destroy = "true"
}