I have used terraform scripts to create resources in GCP. The scripts are working fine. But my question is - how do I create multiple storage buckets using a single script.
I have two files for creating the storage bucket-
main.tf which has the terraform code to create the buckets .
variables.tf which has the actual variables like storage bucket name, project_id, etc, which looks like this:
variable "storage_class" { default = "STANDARD" }
variable "name" { default = "internal-demo-bucket-1"}
variable "location" { default = "asia-southeast1" }
How can I provide more than one bucket name in the variable name? I tried to provide multiple names in an array but the build failed.
I don't know all your requirements, however suppose you need to create a few buckets with different names, while all other bucket characteristics are constant for every bucket in the set under discussion.
I would create a variable, i.e. bucket_name_set in a variables.tf file:
variable "bucket_name_set" {
description = "A set of GCS bucket names..."
type = list(string)
}
Then, in the terraform.tfvars file, I would provide unique names for the buckets:
bucket_name_set = [
"some-bucket-name-001",
"some-bucket-name-002",
"some-bucket-name-003",
]
Now, for example, in the main.tf file I can describe the resources:
resource "google_storage_bucket" "my_bucket_set" {
project = "some project id should be here"
for_each = toset(var.bucket_name_set)
name = each.value # note: each.key and each.value are the same for a set
location = "some region should be here"
storage_class = "STANDARD"
force_destroy = true
uniform_bucket_level_access = true
}
Terraform description is here: The for_each Meta-Argument
Terraform description for the GCS bucket is here: google_storage_bucket
Terraform description for input variables is here: Input Variables
Have you considered using terraform provided modules ? It becomes very easy if you use gcs module for bucket creation. It has an option to specify how many buckets you need to create and even the subfolders. I am including the module below for your reference
https://registry.terraform.io/modules/terraform-google-modules/cloud-storage/google/latest
Related
I am very new to terraform. My requirement is to upload objects to existing s3 buckets. I want to upload one or more objects from my source to one or more buckets utilizing only one resource. Using count and count.index I can create different numbers of resources. However, doing so will prevent me from using fileset which helps to recursively upload all the contents in the folder.
The basic code look like this. This is for multiple file uploads to single bucket but I would like to modify for multiple uploads to different buckets.;
variable "source_file_path"{
type = list(string)
description = "Path from where objects are to be uploaded"
}
variable "bucket_name"{
type = list(string)
description = "Name or ARN of the bucket to put the file in"
}
variable "data_folder"{
type = list(string)
description = "Object path inside the bucket"
}
resource "aws_s3_bucket_object" "upload_object"{
for_each = fileset(var.source_file_path, "*")
bucket = var.bucket_name
key = "${var.data_folder}${each.value}"
source = "${var.source_file_path}${each.value}"
}
I have created a vars.tfvars file with following values;
source_file_path = ["source1","source2"]
bucket_name = ["bucket1","bucket2"]
data_folder = ["path1","path2"]
So, what I need is, terraform to be able to upload all the files from the source1 to bucket1 s3 bucket by creating path1 inside the bucket. And similarly for source2, bucket2, and path2.
Is this something that can be done in terraform?
From your problem description it sounds like a more intuitive data structure to describe what you want to create would be a map of objects where the keys are bucket names and the values describe the settings for that bucket:
variable "buckets" {
type = map(object({
source_file_path = string
key_prefix = string
}))
}
When defining the buckets in your .tfvars file this will now appear as a single definition with a complex type:
buckets = {
bucket1 = {
source_file_path = "source1"
key_prefix = "path1"
}
bucket2 = {
source_file_path = "source2"
key_prefix = "path2"
}
}
This data structure has one element for each bucket, so it is suitable to use directly as the for_each for a resource describing the buckets:
resource "aws_s3_bucket" "example" {
for_each = each.buckets
bucket = each.key
# ...
}
There is a pre-existing official module hashicorp/dir/template which already encapsulates the work of finding files under a directory prefix, assigning each one a Content-Type based on its filename suffix, and optionally rendering templates. (You can ignore the template feature if you don't need it, by making your directory only contain static files.)
We need one instance of that module per bucket, because each bucket will have its own directory and thus its own set of files, and so we can use for_each chaining to tell Terraform that each instance of this module is related to one bucket:
module "bucket_files" {
for_each = aws_s3_bucket.example
base_dir = var.buckets[each.key].source_file_path
}
The module documentation shows how to map the result of the module to S3 bucket objects, but that example is for only a single instance of the module. In your case we need an extra step to turn this into a single collection of files across all buckets, which we can do using flatten:
locals {
bucket_files_flat = flatten([
for bucket_name, files_module in module.bucket_files : [
for file_key, file in files_module.files : {
bucket_name = bucket_name
local_key = file_key
remote_key = "${var.buckets[each.key].key_prefix}${file_key}"
source_path = file.source_path
content = file.content
content_type = file.content_type
etag = file.digests.md5
}
]
])
}
resource "aws_s3_bucket_object" "example" {
for_each = {
for bf in local.bucket_files_flat :
"s3://${bf.bucket_name}/${bf.remote_key}" => bf
}
# Now the rest of this is basically the same as
# the hashicorp/dir/template S3 example, but using
# the local.bucket_files_flat structure instead
# of the module result directly.
bucket = each.value.bucket_name
key = each.value.remote_key
content_type = each.value.content_type
# The template_files module guarantees that only one of these two attributes
# will be set for each file, depending on whether it is an in-memory template
# rendering result or a static file on disk.
source = each.value.source_path
content = each.value.content
# Unless the bucket has encryption enabled, the ETag of each object is an
# MD5 hash of that object.
etag = each.value.etag
}
Terraform needs a unique tracking key for each instance of aws_s3_bucket_object.example, and so I just arbitrarily decided to use the s3:// URI convention here, since I expect that's familiar to folks accustomed to working with S3. This means that the resource block will declare instances with addresses like this:
aws_s3_bucket_object.example["s3://bucket1/path1example.txt"]
aws_s3_bucket_object.example["s3://bucket2/path2other_example.txt"]
Because these objects are uniquely identified by their final location in S3, Terraform will understand changes to the files as updates in-place, but any changes to the location as removing an existing object and adding a new one at the same time.
(I replicated the fact that your example just concatenated the path prefix with the filename without any intermediate separator, and so that's why it appears as path1example.txt above and not path1/example.txt. If you want the slash in there, you can add it to the expression which defined remote_key inside local.bucket_files_flat.)
I want access to my AWS Account ID in terraform. I am able to get at it with aws_caller_identity per the documentation. How do I then use the variable I created? In the below case I am trying to use it in an S3 bucket name:
data "aws_caller_identity" "current" {}
output "account_id" {
value = data.aws_caller_identity.current.account_id
}
resource "aws_s3_bucket" "test-bucket" {
bucket = "test-bucket-${account_id}"
}
Trying to use the account_id variable in this way gives me the error A reference to a resource type must be followed by at least one attribute access, specifying the resource name. I expect I'm not calling it correctly?
If you have a
data "aws_caller_identity" "current" {}
then you need to define a local for that value:
locals {
account_id = data.aws_caller_identity.current.account_id
}
and then use it like
output "account_id" {
value = local.account_id
}
resource "aws_s3_bucket" "test-bucket" {
bucket = "test-bucket-${local.account_id}"
}
Terraform resolves the locals based on their dependencies so you can create locals that depend on other locals, on resources, on data blocks, etc.
Any time you create a datasource in terraform , it will export some attributes related to that datasource so that you can reference it somewhere else in your configuration and interpolate it with various ways.
In your case, you are already referencing the value of your account id in output block
So that same way, you can construct the string for the bucket name as follows.
resource "aws_s3_bucket" "test-bucket" {
bucket = "test-bucket-${data.aws_caller_identity.current.account_id}"
}
I would highly recommend you go through the terrraform syntax which can help you better understand the resource, datasource and expressions
https://www.terraform.io/docs/language/expressions/references.html
Is there a way to reference an id value from another object located in a different main.tf file in a different module?
If the two resources or objects are located in the same file you just do this
resource "aws_s3_bucket" "log_bucket" {
bucket = "my-tf-log-bucket"
acl = "log-delivery-write"
}
resource "aws_s3_bucket" "b" {
bucket = "my-tf-test-bucket"
acl = "private"
logging {
target_bucket = aws_s3_bucket.log_bucket.id
target_prefix = "log/"
}
}
You can assign the target_bucket for logging to the id of the bucket in resource "aws_s3_bucket"
Now suppose I have two folders named log_module and s3_module with their respective main.tf files.
The main.tf inside the log_module contains
resource "aws_s3_bucket" "log_bucket" {
bucket = "my-tf-log-bucket"
acl = "log-delivery-write"
}
and the main.tf inside the s3_module contains
resource "aws_s3_bucket" "b" {
bucket = "my-tf-test-bucket"
acl = "private"
logging {
target_bucket = "target-bucket"
target_prefix = "log/"
}
}
How would I assign the id for the bucket in resource "aws_s3_bucket" "log_bucket"
to the target_bucket in the main.tf for the s3_module?
You can use Terraform output values to achieve this functionality.
In your log_module directory, you can create a new file named outputs.tf and define a new output like so:
output "bucket_id" {
value = aws_s3_bucket.log_bucket.id
}
In your s3_module, you would need to create a variables file (e.g., variables.tf) which would be used to assign a value to the target_bucket for the aws_s3_bucket resource.
For example:
variable "target_bucket" {
description = "The name of the bucket that will receive the log objects"
type = string
}
Then you would modify the main.tf file in your s3_module directory like so:
resource "aws_s3_bucket" "b" {
bucket = "my-tf-test-bucket"
acl = "private"
logging {
target_bucket = var.target_bucket
target_prefix = "log/"
}
}
Where the value of target_bucket is derived from var.target_bucket.
You would then have to create a main.tf file in the root of your repository like so:
module "logging" {
source = "/path/to/log_module/directory"
// Any inputs to the module defined here
}
module "s3" {
source = "/path/to/s3_module/directory"
target_bucket = module.logging.bucket_id
}
The main.tf file in the root of the repository creates in implicit dependency between the s3 and logging modules. The s3 module becomes dependent on the logging module because the value of target_bucket uses the output of the logging module, which is the ID of the S3 bucket.
If you're talking about two modules that are contained within a parent configuration (i.e. everything is all within the same state), then #jasonwalsh has the right answer.
If you're talking about two completely separate Terraform configurations with different state files though (i.e. you run terraform apply on them separately), you'll want to use the remote state data source, combined with outputs in the module that is outputing values.
This allows one Terraform configuration to read output values from an entirely separate Terraform configuration, without them being children of the same parent configuration.
I have this main.tf file:
provider "google" {
project = var.projNumber
region = var.regName
zone = var.zoneName
}
resource "google_storage_bucket" "bucket_for_python_application" {
name = "python_bucket_exam"
location = var.regName
force_destroy = true
}
resource "google_storage_bucket_object" "file-hello-py" {
name = "src/hello.py"
source = "app-files/src/hello.py"
bucket = "python_bucket_exam"
}
resource "google_storage_bucket_object" "file-main-py" {
name = "main.py"
source = "app-files/main.py"
bucket = "python_bucket_exam"
}
When executed first time It worked fine, but after terraform destroy and again terraform plan -> terraform apply I've noticed that terraform tries to create object before actually creating a bucket:
Ofc it cant't create object inside something that does'nt exist. Why is that?
You have to create a dependency between your objects and your bucket (see code below). Otherwise, Terraform won't know that it has to create bucket first, and then objects. This is related to how Terraform stores the resources in a directed graph.
resource "google_storage_bucket_object" "file-hello-py" {
name = "src/hello.py"
source = "app-files/src/hello.py"
bucket = google_storage_bucket.bucket_for_python_application.name
}
resource "google_storage_bucket_object" "file-main-py" {
name = "main.py"
source = "app-files/main.py"
bucket = google_storage_bucket.bucket_for_python_application.name
}
By doing this, you declare an implicit order : bucket, then objects. This is equivalent to using depends_on in your google_storage_bucket_objects, but in that particular case I recommend using a reference to your bucket in your objects, rather than using an explicit depends_on.
I was able to create a bucket in an amazon S3 using this link.
I used the following code to create a bucket :
resource "aws_s3_bucket" "b" {
bucket = "my_tf_test_bucket"
acl = "private"
}
Now I wanted to create folders inside the bucket, say Folder1.
I found the link for creating an S3 object. But this has a mandatory parameter source. I am not sure what this value have to , since my intent is to create a folder inside the S3 bucket.
For running terraform on Mac or Linux, the following will do what you want
resource "aws_s3_bucket_object" "folder1" {
bucket = "${aws_s3_bucket.b.id}"
acl = "private"
key = "Folder1/"
source = "/dev/null"
}
If you're on windows you can use an empty file.
While folks will be pedantic about s3 not having folders, there are a number of operations where having an object placeholder for a key prefix (otherwise called a folder) make life easier. Like s3 sync for example.
Actually, there is a canonical way to create it, without being OS dependent, by inspecting the Network on a UI put you see the content headers, as stated by : https://stackoverflow.com/users/1554386/alastair-mccormack ,
And S3 does support folders these days as visible from the UI.
So this is how you can achieve it:
resource "aws_s3_bucket_object" "base_folder" {
bucket = "${aws_s3_bucket.default.id}"
acl = "private"
key = "${var.named_folder}/"
content_type = "application/x-directory"
kms_key_id = "key_arn_if_used"
}
Please notice the trailing slash otherwise it creates an empty file
Above has been used with a Windows OS to successfully create a folder using terraform s3_bucket_object.
The answers here are outdated, it's now definitely possible to create an empty folder in S3 via Terraform. Using the aws_s3_object resource, as follows:
resource "aws_s3_bucket" "this_bucket" {
bucket = "demo_bucket"
}
resource "aws_s3_object" "object" {
bucket = aws_s3_bucket.this_bucket.id
key = "demo/directory/"
}
If you don't supply a source for the object then terraform will create an empty directory.
IMPORTANT - Note the trailing slash this will ensure you get a directory and not an empty file
S3 doesn't support folders. Objects can have prefix names with slashes that look like folders, but that's just part of the object name. So there's no way to create a folder in terraform or anything else, because there's no such thing as a folder in S3.
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
http://docs.aws.amazon.com/AWSImportExport/latest/DG/ManipulatingS3KeyNames.html
If you want to pretend, you could create a zero-byte object in the bucket named "Folder1/" but that's not required. You can just create objects with key names like "Folder1/File1" and it will work.
old answer but if you specify the key with the folder (that doesn't exist yet) terraform will create the folder automatically for you
terraform {
backend "s3" {
bucket = "mysql-staging"
key = "rds-mysql-state/terraform.tfstate"
region = "us-west-2"
encrypt = true
}
}
I would like to add to this discussion that you can create a set of empty folders by providing the resource a set of strings:
resource "aws_s3_object" "default_s3_content" {
for_each = var.default_s3_content
bucket = aws_s3_bucket.bucket.id
key = "${each.value}/"
}
where var.default_s3_content is a set of strings:
variable "default_s3_content" {
description = "The default content of the s3 bucket upon creation of the bucket"
type = set(string)
default = ["folder1", "folder2", "folder3", "folder4", "folder5"]
}
v0.12.8 introduces a new fileset() function which can be used in combination with for_each to support this natively :
NEW FEATURES:
lang/funcs: New fileset function, for finding static local files that
match a glob pattern. (#22523)
A sample usage of this function is as follows (from here):
# Given the file structure from the initial issue:
# my-dir
# |- file_1
# |- dir_a
# | |- file_a_1
# | |- file_a_2
# |- dir_b
# | |- file_b_1
# |- dir_c
# And given the expected behavior of the base_s3_key prefix in the initial issue
resource "aws_s3_bucket_object" "example" {
for_each = fileset(path.module, "my-dir/**/file_*")
bucket = aws_s3_bucket.example.id
key = replace(each.value, "my-dir", "base_s3_key")
source = each.value
}
At the time of this writing, v0.12.8 is a day old (Released on 2019-09-04) so the documentation on https://www.terraform.io/docs/providers/aws/r/s3_bucket_object.html does not yet reference it. I am not certain if that's intentional.
As an aside, if you use the above, remember to update/create version.tf in your project like so:
terraform {
required_version = ">= 0.12.8"
}