I am saving terraform state to s3 bucket by this doc: https://www.terraform.io/docs/language/settings/backends/s3.html
But it mentioned that I can't use variables A backend block cannot refer to named values (like input variables, locals, or data source attributes).
terraform {
backend "s3" {
bucket = "mybucket"
key = "path/to/my/key"
region = "us-east-1"
}
}
The problem is that I need to run terraform in different AWS account and regions. My s3 bucket name includes accountId and region. How can I make it work without manually update the configuration file?
I think of using Ansible for this scenario. This may not be the most efficient way to do it or the correct use of Ansible but you can make use of Jinja2 templating. So you can create a Jinja2 template file like below.
terraform.tf.j2
terraform {
backend "s3" {
bucket = "{{ aws-account-name/id }}-{{ aws-region }}-terraform"
key = "path/to/my/key"
region = "us-east-1"
}
}
Then when you create the infrastucture, you can feed necessary values to terraform.tf.j2 file and create terraform.tf file dynamically.
Related
I'm trying to setup a remote Terraform backend to S3. I was able to create the bucket, but I used bucket_prefix instead of bucket to define my bucket name. I did this to ensure code re-usability within my org.
My issue is that I've been having trouble referencing the new bucket in my Terraform back end config. I know that I can hard code the name of the bucket that I created, but I would like to reference the bucket similar to other resources in Terraform.
Would this be possible?
I've included my code below:
#configure terraform to use s3 as the backend
terraform {
backend "s3" {
bucket = "aws_s3_bucket.my-bucket.id"
key = "terraform/terraform.tfstate"
region = "ca-central-1"
}
}
AWS S3 Resource definition
resource "aws_s3_bucket" "my-bucket" {
bucket_prefix = var.bucket_prefix
acl = var.acl
lifecycle {
prevent_destroy = true
}
versioning {
enabled = var.versioning
}
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = var.sse_algorithm
}
}
}
}
Terraform needs a valid backend configuration when the initialization steps happens (terraform init), meaning that you have to have an existing bucket before being able to provision any resources (before the first terraform apply).
If you do a terraform init with a bucket name which does not exist, you get this error:
The referenced S3 bucket must have been previously created. If the S3 bucket
│ was created within the last minute, please wait for a minute or two and try
│ again.
This is self explanatory. It is not really possible to have the S3 bucket used for backend and also defined as a Terraform resource. While certainly you can use terraform import to import an existing bucket into the state, I would NOT recommend importing the backend bucket.
Is there a way for Terraform to make changes to an existing S3 bucket without affecting the creation or deletion of the bucket?
For example, I want to use Terraform to enable S3 replication across several AWS accounts. The S3 buckets already exist, and I simply want to enable a replication rule (via a pipeline) without recreating, deleting, or emptying the bucket.
My code looks like this:
data "aws_s3_bucket" "test" {
bucket = "example_bucket"
}
data "aws_iam_role" "s3_replication" {
name = "example_role"
}
resource "aws_s3_bucket" "source" {
bucket = data.aws_s3_bucket.example_bucket.id
versioning {
enabled = true
}
replication_configuration {
role = data.aws_iam_role.example_role.arn
rules {
id = "test"
status = "Enabled"
destination {
bucket = "arn:aws:s3:::dest1"
}
}
rules {
id = "test2"
status = "Enabled"
destination {
bucket = "arn:aws:s3:::dest2"
}
}
}
}
When I try to do it this way, Terraform apply tries to delete the existing bucket and create a new one instead of just updating the configuration. I don't mind trying terraform import, but my concern is that this will destroy the bucket when I run terraform destroy as well. I would like to simply apply and destroy the replication configuration, not the already existing bucket.
I would like to simply apply and destroy the replication configuration, not the already existing bucket.
Sadly, you can't do this. Your bucket must be imported to TF so that it can be managed by it.
I don't mind trying terraform import, but my concern is that this will destroy the bucket when I run terraform destroy as well.
To protect against this, you can use prevent_destroy:
This meta-argument, when set to true, will cause Terraform to reject with an error any plan that would destroy the infrastructure object associated with the resource, as long as the argument remains present in the configuration.
I’m looking for recommendations and help with an issue that I am having with setting up and managing bucket and bucket policy creation for multiple environments and multiple regions within a single environment.
I have 4 AWS accounts (dev, stg, prod1, prod2 which is a copy of prod1). In prod1 we have two kubernetes clusters aws-us-prod1 and aws-eu-prod1. These two clusters are completely independent of one another and they merely serve customers in those regions.
I have an applications running on these two different clusters (aws-us-prod1 and aws-eu-prod1) that need to write content to an S3 bucket. But these two clusters share an AWS account (prod1).
I’m trying to write some terraform resource automation to manage this, and I haven’t been able to variably control what region a bucket gets put in. The latest doc shows that there is a region attribute but it doesn’t work because of how the provider has been implemented with the aws provider region attribute.
What I’d like to do is something like this:
variable "buckets" {
type = map(string) # e.g. buckets='{"a-us-prod1": "us-west-2", "a-eu-prod1":"eu-west-2"}'
}
resource "aws_s3_bucket" "my_buckets" {
for_each = var.buckets
bucket = each.key
region = each.value
}
resource "aws_s3_bucket_policy" "my_buckets_policy" {
for_each = aws_s3_bucket.my_buckets
bucket = each.value.id
policy = ...
}
I’ve tried using multiple providers using aliases, but you can’t programmatically use a provider based on the value of a variable you are iterating over. What’s the proper way to organize this project and resources to accomplish this?
These issues I have come across are related to this:
https://github.com/hashicorp/terraform/issues/3656
https://github.com/terraform-providers/terraform-provider-aws/issues/5999
The region attribute just got removed from the s3_bucket in terraform-provider-aws v3.0.0 from July 31, 2020. Before then you could set the region for a bucket and it would have been respected and the bucket would have been created in that selected region. However that was not how any other resource is managed, it was probably just there because S3 is globally scoped and the bucket has no region in the arn. All other services use the region of the provider itself (as it should be).
I would recommend to create the different providers for all the different regions you may want to support and then splitting the var.buckets according to their region and then create one resource "aws_s3_bucket" "this_region" { } for each region:
variable "buckets" {
type = map(string) # e.g. buckets='{"a-us-prod1": "us-west-2", "a-eu-prod1":"eu-west-2"}'
}
provider "aws" {
region = "eu-west-2"
alias = "eu-west-2"
}
locals {
eu_west_2_buckets = [for name, region in var.buckets: name if region == "eu-west-2"]
}
resource "aws_s3_bucket" "eu_west_2_buckets" {
count = length(local.eu_west_2_buckets)
bucket = eu_west_2_buckets[count.index]
provider = aws.eu-west-2
}
If you want to only rollout the buckets that match the current deployment region you can do that by simply changing the bucket filtering logic:
variable "buckets" {
type = map(string) # e.g. buckets='{"a-us-prod1": "us-west-2", "a-eu-prod1":"eu-west-2"}'
}
locals {
buckets = [for name, region in var.buckets: name if region == data.aws_region.current.name]
}
data "aws_region" "current" { }
resource "aws_s3_bucket" "buckets" {
count = length(local.buckets)
bucket = buckets[count.index]
}
I was trying to create a remote backend for my S3 bucket.
provider "aws" {
version = "1.36.0"
profile = "tasdik"
region = "ap-south-1"
}
terraform {
backend "s3" {
bucket = "ops-bucket"
key = "aws/ap-south-1/homelab/s3/terraform.tfstate"
region = "ap-south-1"
}
}
resource "aws_s3_bucket" "ops-bucket" {
bucket = "ops-bucket"
acl = "private"
versioning {
enabled = true
}
lifecycle {
prevent_destroy = true
}
tags {
Name = "ops-bucket"
Environmet = "devel"
}
}
I haven't applied anything yet, the bucket is not present as of now. So, terraform asks me to do an init. But when I try to do so, I get a
$ terraform init
Initializing the backend...
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Error loading state: BucketRegionError: incorrect region, the bucket is not in 'ap-south-1' region
status code: 301, request id: , host id:
Terraform will initialise any state configuration before any other actions such as a plan or apply. Thus you can't have the creation of the S3 bucket for your state to be stored in be defined at the same time as you defining the state backend.
Terraform also won't create an S3 bucket for you to put your state in, you must create this ahead of time.
You can either do this outside of Terraform such as with the AWS CLI:
aws s3api create-bucket --bucket "${BUCKET_NAME}" --region "${BUCKET_REGION}" \
--create-bucket-configuration LocationConstraint="${BUCKET_REGION}"
or you could create it via Terraform as you are trying to do so but use local state for creating the bucket on the first apply and then add the state configuration and re-init to get Terraform to migrate the state to your new S3 bucket.
As for the error message, S3 bucket names are globally unique across all regions and all AWS accounts. The error message is telling you that it ran the GetBucketLocation call but couldn't find a bucket in ap-south-1. When creating your buckets I recommend making sure they are likely to be unique by doing something such as concatenating the account ID and possibly the region name into the bucket name.
Our Terraform layout is such that we run Terraform for many aws (100+) accounts, and save Terraform state file remotely to a central S3 bucket.
The new locking feature sounds useful and wish to implement it but I am unsure if I can make use of a central DynamoDB table in the same account as that of our S3 bucket or do I need to create a DynamoDB table in each of the AWS accounts?
You can use a single DynamoDB table to control the locking for the state file for all of the accounts. This would work even if you had multiple S3 buckets to store state in.
The DynamoDB table is keyed on LockID which is set as a bucketName/path. So as long as you have a unique combination of those you will be fine (you should or you have bigger problems with your state management).
Obviously you will need to set up cross account IAM policies to allow users creating things in one account to be able to manage items in DynamoDB.
To use terraform DynamoDB locking, follow the steps below
1.Create an AWS DynamoDB with terraform to lock the terraform.tfstate.
provider "aws" {
region = "us-east-2"
}
resource "aws_dynamodb_table" "dynamodb-terraform-lock" {
name = "terraform-lock"
hash_key = "LockID"
read_capacity = 20
write_capacity = 20
attribute {
name = "LockID"
type = "S"
}
tags {
Name = "Terraform Lock Table"
}
}
2.Execute terraform to create the DynamoDB table on AWS
terraform apply
Usage Example
1.Use the DynamoDB table to lock terraform.state creation on AWS. As an EC2 example
terraform {
backend "s3" {
bucket = "terraform-s3-tfstate"
region = "us-east-2"
key = "ec2-example/terraform.tfstate"
dynamodb_table = "terraform-lock"
encrypt = true
}
}
provider "aws" {
region = "us-east-2"
}
resource "aws_instance" "ec2-example" {
ami = "ami-a4c7edb2"
instance_type = "t2.micro"
}
The dynamodb_table value must match the name of the DynamoDB table we created.
2.Initialize the terraform S3 and DynamoDB backend
terraform init
3.Execute terraform to create EC2 server
terraform apply
To see the code, go to the
Github DynamoDB Locking Example