I am currently having two (maybe conflicting) S3 bucket policies, which show a permanent difference on Terraform. Before I show parts of the code, I will try to give an overview of the structure.
I am currently using a module, which:
Takes IAM Role & an S3 Bucket as inputs
Attaches S3 Bucket policy to the inputted role
Attaches S3 Bucket (allowing VPC) policy to the inputted S3 bucket
I have created some code (snippet and not full code) to illustrate how this looks like for the module.
The policies look like:
# S3 Policy to be attached to the ROLE
data "aws_iam_policy_document" "foo_iam_s3_policy" {
statement {
effect = "Allow"
resources = ["${data. s3_bucket.s3_bucket.arn}/*"]
actions = ["s3:GetObject", "s3:GetObjectVersion"]
}
statement {
effect = "Allow"
resources = [data.s3_bucket.s3_bucket.arn]
actions = ["s3:*"]
}
}
# VPC Policy to be attached to the BUCKET
data "aws_iam_policy_document" "foo_vpc_policy" {
statement {
sid = "VPCAllow"
effect = "Allow"
resources = [data.s3_bucket.s3_bucket.arn, "${data.s3_bucket.s3_bucket.arn}/*"]
actions = ["s3:GetObject", "s3:GetObjectVersion"]
condition {
test = "StringEquals"
variable = "aws:SourceVpc"
values = [var.foo_vpc]
}
principals {
type = "*"
identifiers = ["*"]
}
}
}
The policy attachments look like:
# Turn policy into a resource to be able to use ARN
resource "aws_iam_policy" "foo_iam_policy_s3" {
name = "foo-s3-${var.s3_bucket_name}"
description = "IAM policy for foo on s3"
policy = data.aws_iam_policy_document.foo_iam_s3_policy.json
}
# Attaches s3 bucket policy to IAM Role
resource "aws_iam_role_policy_attachment" "foo_attach_s3_policy" {
role = data.aws_iam_role.foo_role.name
policy_arn = aws_iam_policy.foo_iam_policy_s3.arn
}
# Attach foo vpc policy to bucket
resource "s3_bucket_policy" "foo_vpc_policy" {
bucket = data.s3_bucket.s3_bucket.id
policy = data.aws_iam_policy_document.foo_vpc_policy.json
}
Now let's step outside of the module, where the S3 bucket (the one I mentioned that will be inputted into the module) is created, and where another policy needs to be attached to it (the S3 bucket). So outside of the module, we:
Provide an S3 bucket to the aforementioned module as input (alongside the IAM Role)
Create a policy to allow some IAM Role to put objects in the aforementioned bucket
Attach the created policy to the bucket
The policy looks like:
# Create policy to allow bar to put objects in the bucket
data "aws_iam_policy_document" "bucket_policy_bar" {
statement {
sid = "Bar IAM access"
effect = "Allow"
resources = [module.s3_bucket.bucket_arn, "${module. s3_bucket.bucket_arn}/*"]
actions = ["s3:PutObject", "s3:GetObject", "s3:ListBucket"]
principals {
type = "AWS"
identifiers = [var.bar_iam]
}
}
}
And its attachment looks like:
# Attach Bar bucket policy
resource "s3_bucket_policy" "attach_s3_bucket_bar_policy" {
bucket = module.s3_bucket.bucket_name
policy = data.aws_iam_policy_document.bucket_policy_bar.json
}
(For more context: Basically foo is a database that needs VPC and s3 attachment to role to operate on the bucket and bar is an external service that needs to write data to the bucket)
What is going wrong
When I try to plan/apply, Terraform shows that there is always change, and shows an overwrite between the S3 bucket policy of bar (bucket_policy_bar) and the VPC policy attached inside the module (foo_vpc_policy).
In fact the error I am getting kind of sounds like what is described here:
The usage of this resource conflicts with the
aws_iam_policy_attachment resource and will permanently show a
difference if both are defined.
But I am attaching policies to S3 and not to a role, so I am not sure if this warning applies to my case.
Why are my policies conflicting? And how can I avoid this conflict?
EDIT:
For clarification, I have a single S3 bucket, to which I need to attach two policies. One that allows VPC access (foo_vpc_policy, which gets created inside the module) and another one (bucket_policy_bar) that allows IAM role to put objects in the bucket
there is always change
That is correct. aws_s3_bucket_policy sets new policy on the bucket. It does not add new statements to it.
Since you are invoking aws_s3_bucket_policy twice for same bucket, first time in module.s3_bucket module, then second time in parent module (I guess), the parent module will simply attempt to set new policy on the bucket. When you perform terraform apply/plan again, the terraform will detect that the policy defined in module.s3_bucket is different, and will try to update it. So you end up basically with a circle, where each apply will change the bucket policy to new one.
I'm not aware of a terraform resource which would allow you to update (i.e. add new statements) to an existing bucket policy. Thus I would try to re-factor your design so that you execute aws_s3_bucket_policy only once with all the statements that you require.
Thanks to the tip from Marcin I was able to resolve the issue by making the attachment of the policy inside the module optional like:
# Attach foo vpc policy to bucket
resource "s3_bucket_policy" "foo_vpc_policy" {
count = var.attach_vpc_policy ? 1 : 0 # Only attach VPC Policy if required
bucket = data.s3_bucket.s3_bucket.id
policy = data.aws_iam_policy_document.foo_vpc_policy.json
}
The policy in all cases has been added as output of the module like:
# Outputting only the statement, as it will later be merged with other policies
output "foo_vpc_policy_json" {
description = "VPC Allow policy json (to be later merged with other policies that relate to the bucket outside of the module)"
value = data.aws_iam_policy_document.foo_vpc_policy.json
}
For the cases when it was needed to defer the attachment of the policy (wait to attach it together with another policy), I in-lined the poliicy via source_json)
data "aws_iam_policy_document" "bucket_policy_bar" {
# Adding the VPC Policy JSON as a base for this Policy (docs: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document)
source_json = module.foor_.foo_vpc_policy_json # here we add the statement that has
statement {
sid = "Bar IAM access"
effect = "Allow"
resources = [module.s3_bucket_data.bucket_arn, "${module.s3_bucket_data.bucket_arn}/*"]
actions = ["s3:PutObject", "s3:GetObject", "s3:ListBucket"]
principals {
type = "AWS"
identifiers = [var.bar_iam]
}
}
}
Related
I am getting following error while Setting up a kinesis data firehose event destination for Amazon SES event publishing using terraform. It seems like the terraform created the IAM role but throwing the error while creating the firehose event destination with IAM role.
Whereas able to attach same IAM role with firehose event destination from AWS console which was created by terraform.
If I manually create the same IAM role using AWS console and then pass the ARN of the role to the terraform it works. However if I try to create the role using terraform and then create the event destination it doesn’t work. Can someone pls help me on this.
Error creating SES configuration set event destination: InvalidFirehoseDestination: Could not assume IAM role <arn:aws:iam::<AWS account name >:role/<AWS IAM ROLE NAME>>.
data "aws_iam_policy_document" "ses_configuration_set_assume_role" {
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ses.amazonaws.com"]
}
}
}
data "aws_iam_policy_document" "ses_firehose_destination_policy" {
statement {
effect = "Allow"
actions = [
"firehose:PutRecord",
"firehose:PutRecordBatch"
]
resources = [
"<ARN OF AWS FIREHOSE DELIVERY STREAM >"
]
}
}
resource "aws_iam_policy" "ses_firehose_destination_iam_policy" {
name = "SesfirehosedestinationPolicy"
policy = data.aws_iam_policy_document.ses_firehose_destination_policy.json
}
resource "aws_iam_role" "ses_firehose_destination_role" {
name = "SesfirehosedestinationRole"
assume_role_policy = data.aws_iam_policy_document.ses_configuration_set_assume_role.json
}
resource "aws_iam_role_policy_attachment" "ses_firehose_destination_role_att" {
role = aws_iam_role.ses_firehose_destination_role.name
policy_arn = aws_iam_policy.ses_firehose_destination_iam_policy.arn
}
resource "aws_ses_configuration_set" "create_ses_configuration_set" {
name = var.ses_config_set_name
}
resource "aws_ses_event_destination" "ses_firehose_destination" {
name = "event-destination-kinesis"
configuration_set_name = aws_ses_configuration_set.create_ses_configuration_set.name
enabled = true
matching_types = ["send", "reject", "bounce", "complaint", "delivery", "renderingFailure"]
depends_on = [aws_iam_role.ses_firehose_destination_role]
kinesis_destination {
stream_arn = "<ARN OF AWS FIREHOSE DELIVERY STREAM>"
role_arn = aws_iam_role.ses_firehose_destination_role.arn
}
}
You might need to look at your Firehose datasource. If it is a Kinesis Datastream, it will not work. It will only work when using a Direct PUT and other datasource for the Kinesis Firehose. I ran into this issue while setting this up for my Kinesis Firehose to Datadog as well. I hope that this helps.
I found the same issue and was able to resolve it with a slight workaround.
The issue is likely due to the time it takes AWS to propagate the IAM role to all the regions. As an IAM role is global, it will be created first in a 'random' region and then propagated to all regions. So if it is not first created in your region it may take some time to propagate and you will get this error if the SES Event Destination is created before the IAM role has propagated.
It does not help to add a depends_on clause, as terraform (correctly?) thinks the IAM role has been created, it has just not been propagated to your region yet.
The solution that worked for me was to create an IAM role that grants access to the "sts:AssumeRole" action for the SES service and the "firehose:PutRecordBatch" action for the Firehose. When I apply the Terraform, I did a targeted apply first for only this role, waited a minute (to allow the IAM role to propagate) and then do a normal terraform apply to complete.
In your example the command will look something like this:
terraform apply --target aws_iam_role_policy_attachment.ses_firehose_destination_role_att
terraform apply
I'm trying to setup a remote Terraform backend to S3. I was able to create the bucket, but I used bucket_prefix instead of bucket to define my bucket name. I did this to ensure code re-usability within my org.
My issue is that I've been having trouble referencing the new bucket in my Terraform back end config. I know that I can hard code the name of the bucket that I created, but I would like to reference the bucket similar to other resources in Terraform.
Would this be possible?
I've included my code below:
#configure terraform to use s3 as the backend
terraform {
backend "s3" {
bucket = "aws_s3_bucket.my-bucket.id"
key = "terraform/terraform.tfstate"
region = "ca-central-1"
}
}
AWS S3 Resource definition
resource "aws_s3_bucket" "my-bucket" {
bucket_prefix = var.bucket_prefix
acl = var.acl
lifecycle {
prevent_destroy = true
}
versioning {
enabled = var.versioning
}
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = var.sse_algorithm
}
}
}
}
Terraform needs a valid backend configuration when the initialization steps happens (terraform init), meaning that you have to have an existing bucket before being able to provision any resources (before the first terraform apply).
If you do a terraform init with a bucket name which does not exist, you get this error:
The referenced S3 bucket must have been previously created. If the S3 bucket
│ was created within the last minute, please wait for a minute or two and try
│ again.
This is self explanatory. It is not really possible to have the S3 bucket used for backend and also defined as a Terraform resource. While certainly you can use terraform import to import an existing bucket into the state, I would NOT recommend importing the backend bucket.
My objective is to be able to set up an IAM Role which can assume a role of a certain IAM user. After the creation of the role, I would like to come back later and modify this role by adding external IDs to establish a trust relationship. Let me illustrate with an example:
Let's say I want to create role:
resource "aws_iam_role" "happy_role" {
name = "happy-role"
assume_role_policy = data.aws_iam_policy_document.happy_assume_rule_policy.json
}
Let's also assume that happy_assume_role_policy looks something like:
data "aws_iam_policy_document" "happy_assume_role_policy" {
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
principals {
type = "AWS"
identifiers = [var.some_iam_user_arn]
}
}
}
Now, I will use the created role to create an external integration. But once I am done creating that integration, I want to go back to the role I originally created and modify it's assumed role policy. So now I want to add a condition to the assume role policy and make it look like:
data "aws_iam_policy_document" "happy_assume_role_policy" {
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
principals {
type = "AWS"
identifiers = [var.snowflake_iam_user_arn]
}
condition {
test = "StringEquals"
values = [some_integration.integration.external_id]
variable = "sts:ExternalId"
}
}
}
In other words, my workflow should be like:
Create role without assume conditions
Create an integration with that role
Take the ID from the created integration and go back to the created role and add a condition on it
Edit:
By "integration" I mean something like this. Once an Integration is created, there is an outputted ID, and then I need to take that ID and feed it back to the Assume Role I originally created. That should happen everytime I add a new integration.
I first tried to create two IAM roles, one for managing the integration creation, and another for managing the integration itself. That ran without circular reference errors; however, I was not able to establish a connection from the storage to the database, as it needs to be the same IAM Role creating and managing the integration.
This is what I have ended up doing (still not good for an accepted way to do it IMO). I created a role (with targeting) like:
resource "aws_iam_role" "happy_role" {
name = "happy-role"
assume_role_policy = data.aws_iam_policy_document.basic_policy.json
}
And used a basic assume role policy (without conditions). And then for the next run, I applied (without targeting) and it worked.
I followed the approach mentioned here, How to create a Snowflake Storage Integration with AWS S3 with Terraform?
As part of Storage Integration creation, just provide the role arn which is manually constructed without the resource is created. Terraform wont complain. Then create the role with the assume policy referring to the External Id and User ARN created by the Storage Integration
I am trying to create multiple s3 buckets each with different bucket settings. i am Looking for syntax on how to refer the bucket ids of the dynamically created bucket in other bucket resource blocks.
New to terraform. looking for sample code or terrraform document for this syntax
Bel0w is sample code for creating bucket from list names
resource "aws_s3_bucket" "this" {
count=length(var.bucket_names)
bucket = var.bucket_names[count.index]
acl="private"
versioning {
enabled = var.bucket_versioning
}
}
In this code i want to refer the dynamically created bucket id's and assign their bucket policy settings. Need the syntax . not sure if this correct
resource "aws_s3_bucket_policy" "this" {
count=length(var.bucket_names)
bucket = aws_s3_bucket.this.id[count.index]
policy = data.aws_iam_policy_document.this.json
}
In your aws_s3_bucket_policy, instead of
bucket = aws_s3_bucket.this.id[count.index]
it should be
bucket = aws_s3_bucket.this[count.index].id
assuming that everything else is correct, e.g. data.aws_iam_policy_document.this.json is valid.
I want to attach one of the pre-existing AWS managed roles to a policy, here's my current code:
resource "aws_iam_role_policy_attachment" "sto-readonly-role-policy-attach" {
role = "${aws_iam_role.sto-test-role.name}"
policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}
Is there a better way to model the managed policy and then reference it instead of hardcoding the ARN? It just seems like whenever I hardcode ARNs / paths or other stuff like this, I usually find out later there was a better way.
Is there something already existing in Terraform that models managed policies? Or is hardcoding the ARN the "right" way to do it?
The IAM Policy data source is great for this. A data resource is used to describe data or resources that are not actively managed by Terraform, but are referenced by Terraform.
For your example, you would create a data resource for the managed policy as follows:
data "aws_iam_policy" "ReadOnlyAccess" {
arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}
The name of the data source, ReadOnlyAccess in this case, is entirely up to you. For managed policies I use the same name as the policy name for the sake of consistency, but you could just as easily name it readonly if that suits you.
You would then attach the IAM policy to your role as follows:
resource "aws_iam_role_policy_attachment" "sto-readonly-role-policy-attach" {
role = "${aws_iam_role.sto-test-role.name}"
policy_arn = "${data.aws_iam_policy.ReadOnlyAccess.arn}"
}
When using values that Terraform itself doesn't directly manage, you have a few options.
The first, simplest option is to just hard-code the value as you did here. This is a straightforward answer if you expect that the value will never change. Given that these "canned policies" are documented, built-in AWS features they likely fit this criteria.
The second option is to create a Terraform module and hard-code the value into that, and then reference this module from several other modules. This allows you to manage the value centrally and use it many times. A module that contains only outputs is a common pattern for this sort of thing, although you could also choose to make a module that contains an aws_iam_role_policy_attachment resource with the role set from a variable.
The third option is to place the value in some location that Terraform can retrieve values from, such as Consul, and then retrieve it from there using a data source. With only Terraform in play, this ends up being largely equivalent to the second option, though it means Terraform will re-read it on each refresh rather than only when you update the module using terraform init -upgrade, and thus this could be a better option for values that change often.
The fourth option is to use a specialized data source that can read the value directly from the source of truth. Terraform does not currently have a data source for fetching information on AWS managed policies, so this is not an option for your current situation, but can be used to fetch other AWS-defined data such as the AWS IP address ranges, service ARNs, etc.
Which of these is appropriate for a given situation will depend on how commonly the value changes, who manages changes to it, and on the availability of specialized Terraform data sources.
I got a similar situation, and I don't want to use the arn in my terraform script for two reasons,
If we do it on the Web console, we don't really look at the arn, we search the role name, then attach it to the role
The arn is not easy to remember, looks like is not for human
I would rather use the policy name, but not the arn, here is my example
# Get the policy by name
data "aws_iam_policy" "required-policy" {
name = "AmazonS3FullAccess"
}
# Create the role
resource "aws_iam_role" "system-role" {
name = "data-stream-system-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Sid = ""
Principal = {
Service = "ec2.amazonaws.com"
}
},
]
})
}
# Attach the policy to the role
resource "aws_iam_role_policy_attachment" "attach-s3" {
role = aws_iam_role.system-role.name
policy_arn = data.aws_iam_policy.required-policy.arn
}